Internet Of Things (IOT) Big Data Artificial Intelligence Expert System Information Management And Control Systems And Methods

ABSTRACT

IoT Big Data information management and control systems and methods for distributed performance monitoring and critical network fault detection comprising a combination of capabilities including: IoT data collection sensor stations receiving sensor input signals and also connected to monitor units providing communication with other monitor units and/or cloud computing resources via IoT telecommunication links, and wherein a first data collection sensor station has expert predesignated other network elements comprising other data collection sensor stations, monitor units, and/or telecommunications equipment for performance and/or fault monitoring based on criticality to said first data collection sensor station operations, thereby extending monitoring and control operations to include distributed interdependent or critical operations being monitored and analyzed throughout the IoT network, and wherein performance and/or fault monitoring signals received by said first data collection sensor station are analyzed with artificial intelligence, hierarchical expert system algorithms for generation of warning and control signals.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 17/183,653 filed onFeb. 24, 2021, which is a continuation of Ser. No. 16/412,383 filed onMay 14, 2019 entitled IOT Sensor Network Artificial IntelligenceWarning, Control and Monitoring Systems And Methods.

BACKGROUND OF THE INVENTION

The Internet of Things (IoT) is a network of physical devices or objects(“things”) monitored and/or controlled by distributed sensors,controllers, processors and storage devices interconnected by theInternet. The physical devices or objects may include, for example:materials, objects, persons, areas, terrestrial or air-borne vehicles,appliances, manufacturing or process tools, environments, pipe lines,power generation and/or delivery systems, telecommunications equipment,processors and/or storage devices, or other devices or objects for whichcollected information and/or automated control is important forconsiderations such as safety, personal health or well-being, security,operational efficiency, information exchange, data processing and datastorage.

The importance and magnitude of the IoT cannot be overstated. It hasbeen estimated that the number of devices connected to the IoT mayexceed 20 Billion or more by 2020. The total annual revenues for vendorsof hardware, software and IOT solutions has been estimated to exceed$470B by 2020 (See, for example, Louis Columbus, “Roundup of Internet ofThings Forecasts and Market Estimates,” Forbes, Nov. 27, 2016.)Efficient management and control of such massive networks is of criticalimportance. This invention addresses improved performance and operationof such IoT systems and methods providing Artificial Intelligence (AI)integrated and comprehensive overall network operational monitoringsystems and methods. Network sensors, controllers, telecommunicationnetwork resources and processing and data storage resources are includedin the systems and methods of this invention.

A critical concern is management of the massive amounts of datacollected from billions of sensors implemented throughout the IoT.Modern technology is being employed to amass this data in distributedcomputer and data storage systems including “cloud” based systems. Themassive data bases being assembled are often referred to “Big Data.”

Big data has been defined as voluminous and complex data sets. Oftentraditional data-processing application software are inadequate to dealwith Big Data. Challenges include capturing data, data storage, dataanalysis, search, sharing, transfer, visualization, querying, updating,information privacy and data source access. Managing and makingefficient use of such Big Data is a challenge to system designers.

One aspect of the present invention is to provide such Big Dataefficient data management and use systems and methods based onartificial intelligence, expert systems, fuzzy logic and hierarchicaland adaptive expert and fuzzy system implementations. More particularly,the systems and methods disclosed herein provide efficient and powerfulderivation of IoT warning and control signals directed to managing IoTnetwork faults and potentially dangerous situations. It is alsoimportant that information provided be readily understandable andpresented in a non-confusing format. In many cases, such readilyunderstandable presentation may be critical to control and appropriateresponse to IoT situations being monitored.

While managing Big Data collected on centralized servers is important,the rapid advance in distributed processing can also alleviatecentralized processing requirements. The present invention disclosesartificial intelligence systems and methods implemented on a distributednetwork basis for performing local processing operations while alsoproviding access to Big Data and cloud-based processing and datastorage.

Sensors available to monitor operations in IoT networks include, forexample, audio sensors, image sensors, medical sensors, locationsensors, process control sensors and equipment sensors, magneticsensors, micro-switches, proximity sensors, RFID (Radio FrequencyIdentification Devices) touch sensitive devices, force sensors such asstrain gauges, optical sensors, infrared sensors, ultraviolet sensors,taste sensors, and environmental sensors including, for example,temperature sensors, humidity sensors, wind sensors and gas sensors.

An additional important consideration for proper operation of the IoT isthe reliability of the backbone telecommunications network and remotedata processing and storage facilities. Failure or congestion from overloading of these resources may negatively impact proper operation of theIoT resulting in, for example, loss of important information or lack ofproper response to critical situations.

In addition to the above described sensor technology, important advancesin various other technologies are available today to implement morepowerful systems and methods for monitoring and/or control of IoTphysical devices, situations and telecommunication network performance,Such technologies include, for example: advanced microprocessor; digitalcontrol; display technology; cloud computing and storage; computertechnology; data storage software and programming languages; advancedradio signal transmission and reception including Wi-Fi, Bluetooth; nearfield communication (NFC); satellite communications; advancedtelecommunications network technology including fiber optictransmission, switching and routing control systems and specializedantenna systems; drones; robotics and BOTs; audio signal processing andacoustical beamforming; image signal generation and transmission; imagesignal analysis; speech recognition; speech-to-text conversion;text-to-speech conversion; natural language processing; electroniclocation determination; artificial intelligence; expert systems, fuzzylogic; neural networks; statistical signal analysis; network graphtheory and modern control systems and theory. It is important that suchmonitoring systems and methods be accurate and simple to control andoperate.

Monitoring systems and methods are used today to observe activities atremote locations. Such prior art monitoring systems and methods make useof remote monitoring units or sensors strategically placed in the areato be monitored. Remote sensors may include microphones, motion sensors,image sensors, location sensors, environmental sensors, medical sensors,equipment operational sensors, and the like. The signals from thosesensors are transmitted to network monitoring stations.

Exemplary prior art activity monitoring systems and methods and selectedtechnologies include the following:

C. W. Anderson, “Activity monitor,” U.S. Pat. No. 8,743,200, HiPassDesign, Jun. 3, 2014, describing, inter alia, a system for monitoring alocation using a sensor system and detecting and responding to“interesting events.” The system may detect events based on videoprocessing of a substantially live sequence of images from a videocamera or using other sensors. Embodiments with smart phones andwireless networks are described.

Wen and Tran, “Patient Monitoring Apparatus,” U.S. Pat. Nos. 7,420,472and 7,502,498, Oct. 16, 2005 and Mar. 10, 2009, describing, inter alia,a system using one or more cameras to generate a 3D model of a personand to generate alarms based on dangerous situations determined fromthat model. Fuzzy Logic is mentioned. “Once trained, the data receivedby the server 20 can be appropriately scaled and processed by thestatistical analyzer. In addition to statistical analyzers, the server20 can process vital signs using rule-based inference engines, fuzzylogic, as well as conventional if-then logic. Additionally, the servercan process vital signs using Hidden Markov Models (HMMs), dynamic timewarping, or template matching, among others.”

Ryley, et. al., “Wireless motion sensor using infrared illuminator andcamera integrated with wireless telephone,” U.S. Pat. No. 7,339,608,VTech Telecommunications, Mar. 4, 2008, describing, inter alia, amonitor system with a camera using visible or infrared radiation and acordless radio transceiver for transmission of video and image signalsand alerts.

Karen Fitzgerald, et. al., “Two-Way Communication Baby Monitor withSmoothing Unit,” U.S. Pat. No. 6,759,961, Mattel, Jul. 6, 2004,describing, inter alia, a system with baby monitoring and parent unitsfor communicating audible sounds between the baby and parent units andhaving audible sounds for soothing of the baby.

Karen Fitzgerald, et. al., “Baby monitor with a soothing unit,” U.S.Pat. No. 7,049,968, Mattel, May 23, 2006, describing, inter alia, asystem with baby monitoring and parent units for communicating audiblesounds between the baby and parent units and having audible sounds forsoothing of the baby.

Marc R. Matsen, et.al., “Methods and systems for monitoring structuresand systems,” U.S. Pat. No. 7,705,725, Boeing, Apr. 27, 2010,describing, inter alia, methods and systems for structural and componenthealth monitoring of an object such as a physical aircraft with aplurality of sensor systems positioned about an object to be monitoredand a processing system communicatively coupled to at least one of saidplurality of sensor systems with processing system including expertsystems, neural networks, and artificial intelligence technologies.

Bao Tran, “Mesh network personal emergency response appliance,” U.S.Pat. No. 7,733,224, Jun. 8, 2010, describing, inter alia, a monitoringsystem including one or more wireless nodes forming a wireless meshnetwork; a user activity sensor including a wireless mesh transceiveradapted to communicate with the one or more wireless nodes using thewireless mesh network; and a digital monitoring agent coupled to thewireless transceiver through the wireless mesh network to requestassistance from a third party based on the user activity sensor. Thedigital monitoring agent comprises one of: a Hidden Markov Model (HMM)recognizer, a dynamic time warp (DTW) recognizer, a neural network, afuzzy logic engine, a Bayesian network, an expert system or arule-driven system.

Edward K. Y. Jung, et. al., “Occurrence data detection and storage formote networks,” U.S. Pat. No. 8,275,824, The Invention Science Fund,Sep. 25, 2012, describing, inter alia, systems and processes fordetecting and storing occurrence data using mote networks. Artificialintelligence with pattern recognition may include data or imageprocessing and vision using fuzzy logic, artificial neural networks,genetic algorithms, rough sets, and wavelets.

Matthias W. Rath, et. al., “Method and system for real timevisualization of individual health condition on a mobile device,” U.S.Pat. No. 9,101,334, Aug. 11, 2015, describing, inter alia, a method andtechnology to display 3D graphical output for a user using body sensordata, personal medical data in real time with expert Q&As, “What if”scenarios and future emulation all in one artificial intelligence expertsystem.

M. Toy, “Systems and methods for managing a network,” U.S. Pat. No.9,215,181, Comcast Cable, Dec. 15, 2015 describing, inter alia, systemsand methods for managing congestion in a network. The use of expertsystems, fuzzy logic and neural networks are mentioned. “The methods andsystems can employ Artificial Intelligence techniques such as machinelearning and iterative learning. Examples of such techniques include,but are not limited to, expert systems, case-based reasoning, Bayesiannetworks, behavior-based AI, neural networks, fuzzy systems,evolutionary computation (e.g. genetic algorithms), swarm intelligence(e.g. ant algorithms), and hybrid intelligent systems (e.g. expertinference rules generated through a neural network or production rulesfrom statistical learning).”

C. M. Chou, et. al., “Network operating system resource coordination,”U.S. Pat. No. 9,807,640, Taiwan semiconductor, Oct. 31, 2017 describing,inter alia, network device coordination schemes to allocate resources,configure transmission policies, and assign users to utilize resources.“Inference can also refer to techniques employed for composinghigher-level events from a set of events and/or data”. Such inferenceresults in the construction of new events or actions from a set ofobserved events and/or stored event data, whether the events arecorrelated in close temporal proximity, and whether the events and datacome from one or several event and data sources. The use of expertsystems, fuzzy logic and neural networks are mentioned. “Variousclassification schemes and/or systems (e.g., support vector machines,neural networks, expert systems, Bayesian belief networks, fuzzy logic,and data fusion engines) can be employed in connection with performingautomatic and/or inferred action in connection with the disclosedsubject matter.”

Yiu L Lee, U.S. Publication No. 2014-0126356, patent application Ser.No. 13/669,039, Nov. 6, 2012, Comcast Cable, “ Intelligent Network,”describing, inter alia, determining a plurality of services to beprovided over a first communication path to a destination, determining aselect service of the plurality of services to be provided over afailover path to the destination, detecting a failure of the firstcommunication path, and routing the select service over the failoverpath in response to the failure of the first communication path. The useof expert systems, fuzzy logic and neural networks are mentioned. “Themethods and systems can employ Artificial Intelligence techniques suchas machine learning and iterative learning. Examples of such techniquesinclude, but are not limited to, expert systems, case-based reasoning,Bayesian networks, behavior-based AI, neural networks, fuzzy systems,evolutionary computation (e.g. genetic algorithms), swarm intelligence(e.g. ant algorithms), and hybrid intelligent systems (e.g. expertinference rules generated through a neural network or production rulesfrom statistical learning).”

Robert C. Streijl, “Enhanced network congestion application programminginterface,” AT&T Intellectual Property, U.S. Publication No.2016-0135077, patent application Ser. No. 14/534,499, Nov. 6, 2014,describing, inter alia, systems and methods that receive network loaddata that provides indication of a utilization level extant in awireless cellular network. The use of expert systems, fuzzy logic andneural networks are mentioned. “Any suitable scheme (e.g., neuralnetworks, expert systems, Bayesian belief networks, support vectormachines (SVMs), Hidden Markov Models (HMMs), fuzzy logic, data fusion,etc.) can be used by detection engine 102 to provide an appropriateprediction to associate with the stability factor.”

J. B. Dowdall, “Methods and systems for pedestrian avoidance”, U.S. Pat.No. 9,336,436, May 10, 2016, describing, inter alia, an autonomousvehicle configured to avoid pedestrians using hierarchical cylindricalfeatures.

J. Zu and P. Morton, “Methods and systems for Pedestrian avoidance usingLidar,” U. S. Pat. No. 9,315,192, Apr. 19, 2016, describing anautonomous vehicle configured to avoid pedestrians using hierarchicalcylindrical features and Lidar or other sensors.

J. Benesty, et. al., “Microphone Array Signal Processing,” Springer,2008 is a text treating fundamentals of microphone arrays andbeamforming technologies.

M. Brandsttein and D. West, “Microphone Arrays,” Springer, 2001, is atext dealing with a microphone array signal processing techniques andapplications.

Ali O. Abid Noor, “Adaptive Noise Cancellation—Innovative Approaches,”Lambert Academic Publishing, 2012 is a text describing noisecancellation systems based on optimized subband adaptive filtering.

J. C. Giarratano, et. al., “Expert Systems,” Thomson Course Technology,Boston, Mass., 2005 is a text dealing with knowledge representation,reasoning modeling and expert system design.

Chen, C. H., “Fuzzy Logic and Neural Network Handbook,” McGraw-Hill, NewYork, 1996.

Cox, C., “The Fuzzy Systems Handbook,” Academic Press Inc., 1994.

Earl Cox, “Fuzzy Fundamentals,” IEEE Spectrum, October 1992, pages58-61. A technical paper describing, inter alia, basic concepts of fuzzylogic and its applications.

G. V. S. Raju, et. al., “Hierarchical Fuzzy Control,” Int J. Control,1991, V. 54, No. 5, pages 1201-1216. A technical paper describing, interalia, use of Hierarchical Fuzzy Logic with rules structured in ahierarchical way to reduce the number or required rules from anexponential function of the system variables to a linear function ofthose variables.

G. V. S. Raju, “Adaptive Hierarchical Fuzzy Controller,” IEEETransactions on Systems, Man and Cybernetics, V. 23, No. 4, pages973-980, July/August 1993. A technical paper describing, inter alia, useof a supervisory rule set to adjust the parameters of a hierarchicalrule-based fuzzy controller to improve performance.

Li-Xin Wang, “Analysis and Design of Hierarchical Fuzzy Systems,” IEEETransactions on Fuzzy Systems, V. 7, No. 5, October 1999, pages 617-624.A technical paper describing, inter alia, derivation of a gradientdecent algorithm for tuning parameters of hierarchical fuzzy system tomatch input-output pairs.

Di Wang, Xiao-Jun Zeng and John A. Keane, “A Survey of HierarchicalFuzzy Systems (Invited Paper),” International Journal of ComputationalCognition, V. 4, No. 1, 2006, pages 18-29. A technical paper providing asurvey of fuzzy hierarchical systems.

S. Bolognani and M. ZiglIoTto, “Hardware and Software EffectiveConfigurations for Multi-Input Fuzzy Logic Controllers,” IEEETransactions on Fuzzy Systems, V. 6, No. 1, February 1998, pages173-179. A technical paper describing, inter alia, approaches tosimplification of multiple input fuzzy logic controllers with either ahierarchical or parallel structure.

F. Cheong and R. Lai, “Designing a Hierarchical Fuzzy Controller UsingDifferential Evolution,” IEEE International Fuzzy Systems ConferenceProceedings, Seoul Korea, August 22-25, 1999, pages 1-277 to 1-282. Atechnical paper describing, inter alia, a method for automatic design ofa hierarchical fuzzy logic controllers.

Elike Hodo, et, al., “Threat analysis of IoT networks using artificialneural network intrusion detection system,” International Symposium onNetworks, Computers and Communications (ISNCC), May 2016. A technicalpaper describing, inter alia, a threat analysis of the IoT and uses anArtificial Neural Network (ANN) to combat these threats.

Xiaoyu Sun, et. al., “Low-VDD Operation of SRAM Synaptic Array forImplementing Ternary Neural Network,” IEEE Transactions on very LargeScale Integration (VLSI) systems, V. 25, No. 10, October, 2017, pages262-265. A technical paper describing, inter alia, a low-power design ofa static random access memory (SRAM) synaptic array for implementing alow-precision ternary neural network.

E. De Coninck, et. al., “Distributed Neural Networks for Internet ofThings: The Big-Little Approach,” from book Internet of Things—IoTInfrastructures: Second International Summit, IoT 360°, Rome, Italy,Oct. 27-29, 2015, pp.484-492. A technical paper describing, inter alia,an application area in the Internet of Things (IoT) where a massiveamount of sensor data has to be classified and the need to overcomevariable latency issues imposes a major drawback for neural networks.The paper describes a proposed Big-Little architecture with deep neuralnetworks used in the IoT.

F. Chung and J. Duan, “On Multistage Fuzzy Neural Network Modeling,”IEEE Transactions on Fuzzy Systems, Vol. 8, No. 2, Apr. 2000, pages125-142. A technical paper addressing, inter alia, input selection formultistage hierarchical AI network models and proposed efficient methodsof selection.

M. Chi, et.al., “Big Data for Remote Sensing: Challenges andOpportunities,” IEEE Proceedings, Vl. 104, No. 11, November 2016, pages2207-2219.

M. Frustaci et. al., “Evaluating Critical Security Issues of the IoTWorld: Present and future challenges,” IEEE Internet of Things Journal,August, 2018, pages 2483-2495

However, such prior systems and methods fail to take full advantage ofmodern AI expert system, fuzzy logic, neural network and hierarchicalsystem information processing technology to provide a comprehensiveassessment of sensor data, telecommunication network status, and/orpotentially dangerous or unacceptable situations or conditions. What isneeded are newly improved monitoring systems and methods that analyzeand integrate information from multiple network sensors includingphysical device sensors, situation sensors, distributed sensors inremote locations and telecommunication network problem sensors togenerate integrated, understandable and non-confusing assessments forpresentation to monitoring personnel and/or control systems.

SUMMARY OF INVENTION

Various embodiments for improved monitoring systems and methods aredisclosed in the present invention. In one aspect of this invention, afirst Internet of Things (IoT) sensor network remote sensor stationcomprises, without limitation, a sensor network parameter processing,warning and control system with at least one electronic, specificallyprogrammed, specialized sensor network communication computer machineincluding electronic artificial intelligence expert system processingand further comprising a non-transient memory having at least oneportion for storing data and at least one portion for storing particularcomputer executable program code; at least one processor for executingthe particular program code stored in the memory; and one or moretransceivers and/or electrical or optical connections for communicatingwith IoT (Internet of Things) sensors that generate electrical oroptical parameter signals derived from sensor inputs from objects orsituations being monitored.

Some embodiments further comprise one or more other different followedor following Internet of Things (IoT) sensor network remote sensorstations sharing common interests with said first IoT sensor networkremote sensor station comprising one or more other electronic,specifically programmed, specialized sensor network communicationcomputer machines for monitoring other such electrical or optical sensorparameter signals derived from different sensor inputs from IoT objectsor situations being monitored.

Some embodiments further comprise one or more monitor units connectedto, collecting information from and communicating with said first remotesensor station and further analyzing such collected information fromremote sensor stations.

Furthermore, in some embodiments, the particular program code mat beconfigured to perform artificial intelligence expert system operationsupon execution including artificial intelligence expert systemprocessing based on expert input defining multiple expert system logicpropositional instructions and multiple ranges of sensor variables withartificial intelligence expert system processing analysis of multiplesensor signal inputs and generation of multiple control outputs withurgent and/or integrated composite degree of concerns based on saidexpert system propositional instruction evaluation of multiple inputsensor parameters.

In some embodiments, the artificial intelligence expert systemprocessing further comprises hierarchical Multiple-Input/Multiple-Output(MIMO) operation wherein the number of said expert system logicpropositional instructions is a linear function of the number ofvariables and wherein said hierarchical MIMO operations provide inputsto successive hierarchical control levels based at least in part onimportance of said inputs and feedback indicative of output signalsensitivity to said inputs with artificial intelligence expert systemcontrol of dispatch of electronic or optical communication warningsand/or corrective action to address MIMO urgent concerns and/orcomposite degrees of concern of said sensor network objects orsituations based on urgent concerns and rankings of said expert systemcomposite degrees of concern.

In some embodiments, the artificial intelligence expert systemprocessing comprises, without limitation, one or more of expert systemprocessing and analysis of said first remote sensor station sensor inputsignals, acoustic signal processing, speech recognition, naturallanguage processing, image processing, fuzzy logic, statisticalanalysis, mathematical analysis and/or neural network analysis.

In some embodiments, the first sensor network remote sensor stationsensor signals include, without limitation, a combination of one or moreof audio, image, medical, process, material, manufacturing equipment,environmental, transportation, location, pipeline, power system,radiation, vehicle, computer, processor, data storage, cloud processing,cloud data storage, vehicle, drone, threat, mote, BOT, robot,telecommunication network, cyberattack, malicious hacking or otherfollowed remote sensor station monitoring signals.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station of said MIMO artificialintelligence expert system controller is a fuzzy logic controller.

In some embodiments, the artificial intelligence expert system remotesensor station of said hierarchical MIMO artificial intelligence expertsystem controller is a fuzzy logic controller.

In some embodiments, the artificial intelligence expert systempropositional expert system instructions are based on priorities orimportance of selected object or situation expert defined monitoredparameters.

In some embodiments, the artificial intelligence expert system includesat least one of said expert systems propositional expert systeminstructions priorities is based on selected combinations of object orsituation parameters.

In some embodiments, the artificial intelligence expert systemprocessing of said first sensor network remote sensor station furthercomprises neural network processing with backward chaining from computedresults to improve future computational results. See, e.g., FIGS. 20 and

271.

In some embodiments, the artificial intelligence expert system of thefirst sensor network remote sensor station further comprises access ofsaid remote sensor station to internet cloud storage and processingunits.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station sensor inputs may vary with time.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station parameter analysis furthercomprises time series analysis of time variable sensor input data.

In some embodiments, the artificial intelligence expert system, thefirst sensor network remote sensor station time series analysis includesregression analysis of time varying sensor signal parameter values.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station electronic, specificallyprogrammed, specialized sensor network communication computer machinecommunicates with other network nodes to monitor connectedtelecommunication network elements, subnetworks or networks for failuresor performance issues impacting said first remote sensor station.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station one or more of said transceiversmay communicate with a terrestrial or air-born vehicle. See, e.g., FIGS.2A and 2B and

124-126 and 134.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station is implemented in a terrestrial orair-born vehicle. See, e.g., FIGS. 2A and 2B and

124-126, 134.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station one or more of said transceiversmay communicate with a drone. See, e.g., FIGS. 2A and 2B and

124 and 125.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station is implemented in a drone. See,e.g., FIGS. 2A and 2B and

125.

The first sensor network remote sensor station of claim 1 wherein theone or more of said transceivers may communicate with a robot. See,e.g., FIGS. 1 and

106 and 109.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station is implemented in a robot. See,e.g., FIG. 1 and

109.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station one or more transceivers maycommunicate with a BOT. See, e.g., FIG. 1 and

106 and 109.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station is implemented in a BOT. See, e.g.,FIG. 1 and

109.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station transceivers may communicate with amote. See, e.g., FIG. 1 and

106 and 108.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station is implemented in a mote. See,e.g., FIG. 1 and

108.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station monitored objects or situationscomprise one or more persons. See, e.g., FIGS. 1, 2A and 2B and

106, 128, 200 and 228.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station a monitored person is an infant,child, invalid, medical patient, elderly or special needs person. See,e.g., FIGS. 1, 2A and 2B and

106.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station transmits background audio signalsto be broadcast in the area of said person. See, e.g., FIGS. 3, 5, 6, 8,9 and 10A and

142, 164-166, 170-175, 184, 186 and 200.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station transmitted background audiosignals are removed from or attenuated in signals transmitted toconnected monitor units to minimize annoying or unnecessary signalsreceived and/or heard at said monitoring unit while still transmittingaudio signals from the monitored object or person. See, e.g., FIGS. 4-6and

164-166 and 170-177.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station transmits periodic keep-alivesignals to a connected monitor unit to assure users that the remotesensor station is operating correctly. See, e.g., FIGS. 3-6, 8 and 10Aand

142, 164-168,173-177, 184, 186 and 200.

In some embodiments, the artificial intelligence expert system the firstsensor network remote sensor station sensor signals include acombination of at least one telecommunication network sensor inputcombined with other sensor signal inputs. See, e.g., FIG. 2B and

130-132

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station at least one telecommunicationnetwork sensor input is a telecommunication link sensor. See, e.g.,FIGS. 2B, 9, 10B, 11, 18 and 21 and

132, 195, 196, 199, 216-220, 229, 256 and 263.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station comprises at least onetelecommunication network router sensor. See, e.g., FIG. 2B and

219.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station comprises at least onetelecommunication switching system sensor. See, e.g., FIG. 10B and

218.

In some embodiments, the artificial intelligence expert system firstsensor network remote sensor station comprises at least onetelecommunication modem sensor. See, e.g., FIG. 10B and

217 and 218.

BRIEF DESCRIPTION OF THE DRAWINGS

While the invention is amenable to various modifications and alternativeforms, specific embodiments thereof are shown by way of example in thedrawings and will herein be described in detail. The inventions of thisdisclosure are better understood in conjunction with these drawings anddetailed description of the preferred embodiments. The various hardwareand software elements used to carry out the inventions are illustratedin these drawings in the form of diagrams, flowcharts and descriptivetables setting forth aspects of the operations of the invention.

FIG. 1 illustrates, without limitation, an exemplary sensor monitornetwork system comprising a one or more remote sensor stations, one ormore monitoring units and one or more network monitoring centers of thisinvention.

FIG. 2A illustrates, without limitation, an exemplary sensor monitornetwork of this invention.

FIG. 2B illustrates, without limitation, exemplary sensor monitornetwork faults or concerns monitored in this invention.

FIG. 3 illustrates, without limitation, an exemplary remote sensorstation of this invention.

FIG. 4 illustrates, without limitation, an exemplary sensor networkmonitoring unit of this invention.

FIG. 5 illustrates, without limitation, exemplary background audio andkeep alive audio signals of the type used in this invention.

FIG. 6 illustrates, without limitation, exemplary composite audiosignals of the type used in this invention.

FIG. 7 illustrates, without limitation, exemplary acoustic spatialbeamforming for noise cancellation of the type used in this invention.

FIG. 8 illustrates, without limitation, exemplary sensor network monitorcontrols of the type used in this invention.

FIG. 9 illustrates, without limitation, an exemplary sensor monitorsystem flowchart.

FIG. 10A illustrates, without limitation, additional exemplary sensorsignal analysis of FIG. 9.

FIG. 10B illustrates, without limitation, still further additionalexemplary sensor signal analysis of FIG. 9.

FIG. 11 illustrates, without limitation, an exemplary expert systemsensor signal analysis and ranking of this invention.

FIG. 12A illustrates, without limitation, exemplary symmetricaudio-video (AV) expert system rules of this invention.

FIG. 12B illustrates, without limitation, exemplary asymmetricaudio-video (AV) expert system rules of this invention.

FIG. 13 illustrates, without limitation, exemplary fuzzy logicrelationships of this invention.

FIG. 14 illustrates, without limitation, exemplary MIMO hierarchicalexpert system operation.

FIG. 15 illustrates, without limitation, exemplary hierarchicalAV-Medical (AVM) sensor expert system rules with medical TOT sensorsignal analysis of this invention.

FIG. 16 illustrates, without limitation, exemplary hierarchicalAVM-Process (AVMP) sensor expert rules with process TOT sensor signalanalysis of this invention.

FIG. 17 illustrates, without limitation, exemplary hierarchicalAVMP-Following (AVMPF) sensor expert rules with followed IOT remotesensor station signal sensor analysis of this invention.

FIG. 18 illustrates, without limitation, an exemplary hierarchicalAVMPF-Telecommunications (AVMPFT) sensor expert rules with IOTtelecommunication signal sensor analysis of this invention.

FIG. 19 illustrates, without limitation, an exemplary MIMO hierarchicalcomposite warning and control index calculation of this invention.

FIG. 20 illustrates, without limitation, an exemplary neural network ofthis invention.

FIG. 21 illustrates, without limitation, an exemplary sensor networkanalysis diagram.

FIG. 22 illustrates, without limitation, an exemplary sensor networkfuzzy logic inference engine of this invention.

DETAILED DESCRIPTION

The above figures are better understood in connection with the followingdetailed description of the preferred embodiments. Although theembodiments are described in considerable detail below, numerousvariations and modifications will become apparent to those skilled inthe art once the above disclosure is fully appreciated. It is intendedthat the following be interpreted to embrace all such variations andmodifications.

FIG. 1 depicts the IoT monitor system 100 of this invention. The IOTmonitor system 100 comprises the remote sensor station 107 used tomonitor the subject or object 101 and/or its surroundings or definedareas, the monitor unit 102 collecting and processing/analyzinginformation from one or more remote monitor stations 107, and thenetwork monitor center 118 communicating with one or more monitor units102 and/or remote monitor stations 107 with further informationprocessing and analysis of data and information gathered by IoT sensorsdistributed throughout the monitor system 100.

Example objects or subject 101 being monitored may include, withoutlimitation, a person, group of people, physical object or objects, or anarea for surveillance. For example, a person being monitored may be aninfant, invalid, medical patient or special needs person, seniorcitizen, child, prisoner, store customer, criminal, intruder, policeofficer, pedestrian, gathering or crowd, or other person or personsrequiring monitoring or surveillance. Example objects being monitoredmay include, without limitation, manufacturing equipment, transportationequipment, robotic or other material or workpiece processing equipment,BOTs, products, product inventory, terrestrial or air borne vehicles,terrestrial vehicle traffic, pipe lines, utility generation and/ordelivery equipment and systems, valuable assets, and computer and/ordata processing equipment. Example telecommunication equipment mayinclude, without limitations, transmission, switching, routing, datastorage, and telecommunication system and data processing and controlequipment and systems. Example 101 object-subject sensors 114 mayinclude, without limitation, audio sensors, video sensors, movementsensors, environmental sensors, medical sensors, traffic sensors,pedestrian sensors, BOT sensors, robot sensors, mote sensors, processsensors, location sensors, security sensors or other sensors used tomonitor objects or subjects for activities or events of interest. IoTnetwork security concerns include network cyberattacks and malicioushacking of network connected devices and sensors, telecommunicationfacilities, data processing and storage facilities, and informationcollected, stored, transmitted and processed in the IoT network. Theubiquitous deployment of billions of network sensors and IoT connectionof such sensors in a world-wide web presents new threat andvulnerability realities and concerns. Security violations may effectlocal and wider network operations requiring evaluation and responsebeyond isolated limited concerns at a particular attack location.Cyberattacks may occur at IoT network application layers, informationtransmission layers, and data processing layers including internet IoTcloud processing and data storage facilities.

The remote sensor station 107 receives, analyzes, and processes data andinformation from and communicates with the object/subject 101 sensors114. In addition, without limitation, the remote sensor station 107 mayinclude additional sensors 113. Example sensors 113 may include, withoutlimitation, audio sensors such as microphone array 109, image sensors110, process sensors and/or environmental sensors. Sensors 113 may beused for further monitoring of the object or subject 101, or areas oractivities around that object or subject. The remote sensor station 107analyzes object or subject 101 sensor 114 inputs and the sensor 113inputs for detection of various situations that may present danger oroperational issues to the object or subject 101. Audio signals from theobject or situation being monitored may be broadcast from remote sensorstation 107 speaker 111.

In some embodiments, the sensors may be configured as a mote or smallelectronic assembly or a robot capable of performing processing,gathering sensory information and communicating with other connectednodes in the network. The network monitor unit 102 receives sensorsignals from the remote sensor station 107 monitoring the object orsubject 101 and surrounding areas activities.

In some embodiments the remote sensor station may implemented in arobot, a BOT or a mote. In some embodiments the remote sensors may beimplemented in a robot, a BOT or a mote.

The object or subject remote sensor station 107 may include a microphonearray 109 comprising one or more microphones used to listen to soundsfrom the object or subject 101 and the surroundings. In some embodimentsthe microphone array 109 may be replaced by a single omnidirectional ordirectional microphone for listing to the sounds. Advantageously, insome embodiments, microphone array 109 permits the use of microphonebeamforming for enhancing sounds emanating from the subject 101 whilereducing sounds received from directions other than those produced bythe object or subject. As explained more completely below, microphonebeamforming makes use of beamforming software to create acoustical beamswith a main lobe directed at the object or subject that is mostsensitive to sounds originating from the object or subject whileattenuating or perhaps essentially eliminating sounds arriving at themicrophone array from different directions. For example, depending onthe configuration, the beamforming microphone array 109 may be designedto be less sensitive to the broadcast audible signals from speaker 111and more sensitive to sounds emanating from the subject 101. In someembodiments of this invention, it may be desirable to include operatorcontrols that may permit the user of the subject remote sensor station107 to enable or disable microphone array beamforming depending on userpreference. For example, some users may want to hear more clearly allsounds generated by the object or subject 101 or from other sources inthe object or subject's surroundings at selected times or under selectedcircumstances.

The object or subject remote sensor station 107 also may include videoand/or infrared (IR) cameras and an IR illuminator 110 as shown inFIG. 1. In addition, the remote sensor station 107 may include stillimage cameras not shown. The remote sensor station 107 may also includeimage analysis software for analyzing images from the cameras 110 todetermine particular activities of the object or subject 101 and/oractivities in the areas being monitored. Such images may reveal activityof the object or subject 101 such as particular movements even thoughthe object or subject 101 may not be making audible sounds. In addition,dangerous situations absent audible sounds from the object or subject101 may be detected with the cameras or other image collection devices110 including, for example, situations that may indicate a risk to theobject or subject 101 being injured by dangerous activities.

While standard video or still camera technology may be useful inmonitoring object or subject 101 activities in well lightedenvironments, such monitoring with subdued lighting or even in the darkmay not be possible with standard cameras. The use of an infrared IRcamera and infrared IR illuminator as depicted in FIG. 1 permits visualmonitoring even in such difficult lighting situations. The IRilluminator may bathe the area with infrared radiation or structured IRillumination may be used to simplify image analysis for determination ofactivities of the object or subject 101. Such structured lighting mayinclude patterns comprising multiple dots of light in variousarrangements, circular patterns of infrared lighting, straight lines ofinfrared lighting or other patterns most suitable for the chosenenvironment and purpose. IR detectors 110 may also be used as motiondetectors including such use in security systems.

The remote monitor station 107 may also include additional controlsuseful for selecting operational modes of the remote sensor station 107.In some embodiments, sensors 114 and/or 113 may be used for sensing ofother conditions such as unacceptable temperature ranges, airpollutants, dangerous gases, fires or smoke. The additional controls maybe used to turn the remote sensor station 107 on or off, adjust thevolume from speaker 111, select or deselect microphone array 109beamforming, and select appropriate video, still image, or infraredvisual monitoring of the object or subject 101 and the surrounding area.

In some embodiments the remote sensor station 107 may also include adisplay not shown for displaying captured video, still or infraredimages from the cameras 110. In some embodiments the display may also beused for touchscreen control of the subject remote sensor station 107 asdescribed further below.

The remote sensor station 107 may include a radio transceiver such as aBluetooth, Wi-Fi or cellular telephone transceiver or other RFtelecommunication capability known to those of skill in the art forcommunicating with other elements of the network monitor system 100 asdepicted in FIG. 1. The antenna 108 may be used for such RFcommunication of audible, visual and/or control and status information.

The network monitoring system 100 of FIG. 1 may also include monitorunit 102 for remotely monitoring activities of the subject 101. Themonitoring unit 102 includes an antenna 106 for receiving RF signalsfrom the remote sensor station 107. The network monitor unit 102 alsoincludes an RF transceiver compatible with the transceiver of the remotesensor station for receiving RF signals from that unit. For example, theRF transceiver for the monitor unit 102 may include Bluetooth, Wi-Fi,cellular telephone transceiver or other RF, fiber optic or wiredconnections for communicating with other elements of the monitor system100 as depicted in FIG. 1.

The remote monitor unit 102 may further include a display 105 fordisplaying video images received from the remote sensor station 107 ofFIG. 1. The display 105 may also be a touchscreen display for display ofcontrol icons to simplify overall operation as described further below.

Remote network monitor unit 102 may also include speaker 103 used tobroadcast audio signals received from the remote sensor station 107 andused for listening to audible sounds from the object or subject 101 andthe surrounding area. As explained further below, the remote networkmonitor unit 102 may also include video and/or audio processing softwareto further enhance received signals to improve observations of object orsubject 101 activities.

The remote monitor unit 102 may also include controls 104 forcontrolling monitoring of object or subject activities. Such controlsmay turn the network monitor unit 102 on and off, adjust speaker 103volumes, adjust display 105 parameters, and/or select operational modesincluding further control for reduction of background noise in thereceived audio signal.

As depicted in FIG. 1, some embodiments may include RF connections viaantenna 106 or landline connections via wire, cable or fiber optic linksto network monitor center 118. The network monitor center 118 mayreceive periodic status update reports from the network monitor unit 102and/or the object or remote sensor station 107 for recording, analysis,reporting and/or preparation of history files for access by users. Themonitor center 118 may also receive emergency alert signals from themonitor unit 102 and/or the remote sensor station 107 indicatingemergency situations requiring immediate attention. In response to suchemergency alert signals, the subject monitor center 118 may dispatchemergency personnel, contact responsible personnel, sound alarms orotherwise provide or initiate appropriate emergency assistance. Themonitor center 118 may contain communications and computer equipment foruse by specialized personnel trained to respond appropriately toreported emergency situations. Communications may be by wireline, cable,fiber-optics and/or via radio signals with antenna 119.

As also indicated in FIG. 1, the monitoring system 100 may communicatewith the cellular telephone 115 for the purpose of providing informationconcerning the status of object or subject 101 and alarm signalsindicating dangerous situations. In addition, any and all audio/videosignals transmitted using cellular telephone frequencies, Wi-Fi, and/orBluetooth or other telecommunication signals to or from the monitor unit102, the remote sensor station 107 or the monitor station 118 of FIG. 1or transmitted to the cellular telephone 115. The cellular telephone 115also includes one or more antennae 117 for such communications. Thecellular telephone 115 also includes touchscreen display 116 and othercontrols as may be implemented depending upon the cellular telephonedesign. It is to be understood that the cellular telephone 115 may alsobe a tablet computer or laptop computer or other similar portabletelecommunication devices. In some embodiments the remote sensor station107 may in fact be implemented using cellular telephone or portablecomputer or tablet devices.

In addition, any of the monitor unit 102, remote sensor station 107 ormonitor center 118 may transmit email or text messages to designateduser World Wide Web addresses such as personal computers or otherdesignated recipients of information concerning the object or subject101 being monitored. Such communications may make use of SMS (shortmessage service) or other messaging telecommunications with unsolicitedmessages concerning the object or subject 101 being “pushed” to theaddress destination without first being requested from that destination.Such messages enable rapid alerting of designated destinationsconcerning the status of the object or subject 101. Other social mediacommunication systems such as Twitter and Facebook may also be used tocommunicate status information concerning the subject 101 on a real-timeor near real-time basis.

In some embodiments, the monitor system 100 may also communicate withcloud computing 120 as shown in FIG. 1. Cloud computing 120 may comprisecomputing, data processing and data storage equipment useful tosupplement the computing, data processing, and data storage equipment ofthe monitor system 100 of FIG. 1. Cloud computing 120 may compriseextensive data base information descriptive of local, wide area, or evenglobal IoT network performance parameters useful in evaluatingsituations impacting the object or subject 101.

FIG. 2A illustrates an exemplary sensor monitor network 200 of the typeused in some embodiments of the present invention. The sensor monitornetwork 200 may comprise exemplary star subnetworks 201 and 205 and/orexemplary mesh subnetworks 206 interconnected with each other and withthe monitor center 207. The monitor center 207 of FIG. 2A corresponds tothe monitor center 118 FIG. 1. The remote sensor stations 203 of FIG. 2Acorrespond to the remote sensor stations 107 of FIG. 1. The monitor unit202 of FIG. 2A corresponds to the monitor unit 102 of FIG. 1. As shownin FIG. 2A, multiple remote sensor stations 203 may be connected to amonitor unit 202 in star subnetworks such as 201 and 205 or exemplarymesh subnetworks such as 206. Communication links 204 interconnectremote sensor stations 203 to the respective monitor units 202.Communication links 204 may be wireline, fiber optic or RF radio linksas appropriate for a given implementation. Communication links 208in-turn interconnect the monitor units 202 with the monitor center 207.The individual subnetworks 202, 205 and 206 communicate with the monitorcenter 207 and with each other through the monitor center 207 and/or viadirect connections such as shown in link 209 or via cloud computing 212.The communication connections between the exemplary sensor networkremote sensor stations, monitor units and the monitor center may also beestablished via drones such as drone 210 or other air-born vehicle notshown using radio-link RF communications.

Air-born drone 210 may be used to relay signals between remote sensorstations 203 and monitor units 202 of FIG. 2A. The drone 210 may alsorelay signals to and from the monitor center 207 and/or the monitorcenter 118 of FIG. 1. In addition, the drone 210 may itself containsensors such as cameras, temperature, wind, rain, snow, humidity orother sensors suitable for drone implementation. Drone 210 may alsocontain signal analysis software for proper operation with RF linkprotocols, noise reduction and/or data compression as appropriate forparticular applications. In some embodiments, drones such as drone 210may have the functionality of a remote monitor station such as remotemonitor station 203.

Terrestrial vehicle 211 may also communicate with the remote sensorstation 203 via radio links as indicated in FIG. 2A. Vehicle 211 mayalso communicate directly with the monitor unit 202 or network monitorcenter 207. In some embodiments vehicle 211 may be a driverless vehicle.Driverless vehicles communicate with other vehicles in their immediatevicinity to avoid collisions or other issues resulting from trafficcongestion. As indicated in FIG. 2A, a driverless vehicle 211 may bealerted as to the presence of pedestrian 213 in FIG. 2A for the purposeof avoiding hitting that person with vehicle 211. Detection of thepresence of pedestrian 213 may be via radar, sonar, lidar, or via aradio link. The pedestrian may include location determination capabilitysuch as GPS to determine physical location information. That informationmay be transmitted in a format that permits vehicles such as vehicle 211to determine if a collision with the pedestrian may occur permittingcollision avoidance maneuvers or actions to be taken. Warning signalsmay also be transmitted from vehicle 211 to pedestrian 213. In someembodiments such warning signals may be transmitted to the pedestrian213 bringing potentially dangerous situations to the pedestrian's 213attention. In some embodiments, the vehicle 212 may transmit informationon its location, speed and direction of travel. Upon receiving thatinformation, the pedestrian 213 may be alerted of potential collisiondanger via a wireless warning device carried by that pedestrian. In someembodiments, all of the above described pedestrian communication,location determination, processing, signal generation and warning may beimplemented in a pedestrian carried cellular telephone or other portablewireless electronic device carried by the pedestrian.

FIG. 2B depicts the exemplary sensor network of FIG. 2A together withthe exemplary performance or sensor detected issues with differentelements of that network. Exemplary network elements failure or sensordetected issues are illustrated inside dotted circles shown in FIG. 2B.

For example, issue 214 concerns one of the remote sensor stations 203 ofFIG. 2B. The issue 214 may concern complete or partial failure of aremote sensor station 203. The issue 214 may also reflect the outputs ofsensor units being monitored by the remote sensor station 203. Forexample, such sensor outputs may be indicative of alarm signals receivedfrom audio or video sensors. Other possibilities include alarm signalsreceived from medical sensors, environmental sensors or other sensorsused to report the status of objects or situations being monitored bythe remote sensor station 203.

In a similar manner, issue 215 may concern complete or partial failureof the monitor unit 202. Issue 215 may also reflect a reported failureor concerns detected by the multiple remote sensor stations 203connected to monitor unit 202.

Issue 216 concerns the failure or other issues encountered on atelecommunication link 208 connecting remote sensor stations 203 to themonitor center 207. Such issues may include transmission link equipmentproblems, excessive error rates of the transmission link, link datacongestion concerns caused by overloading of the link 208 or propagationissues such as multipath or RF interference.

Issue 217 may concern complete or partial failure of monitor center 207.Issue 217 may indicate concerns about the capacity of monitor center 207to provide required processing and network control. Issue 217 may alsoconcern wider network failures or issues detected by the network monitorcenter 207.

Issue 218 concerns the failure or other issues encountered ontelecommunication links interconnecting monitor units in separatesubnetworks as illustrated in FIG. 2B. Here again, such issues mayinclude transmission link equipment problems, excessive error rates ofthe transmission link, link data congestion concern caused byoverloading of the link or propagation issues such as multipath or RFinterference.

Issue 219 reflects concerns about an entire subnetwork such as theexemplary subnetwork 206. The Issue 219 concerns may range from totalloss of that portion of the sensor network to less drastic issuesinvolving performance of the subnetwork. An example may be the loss ofseveral but not necessarily all elements of mesh subnetwork 206. Anotherexample may be reported sensor issues indicating subnetwork wideproblems.

Issue 220 relates to reported operational concerns for the vehicle 211.Such issues may include, for example, driver issues, equipment issues,roadway issues, traffic issues, emergency accident or police issues,etc. In some embodiments, a complete remote monitor station may beimplemented in vehicle 211. Issue 222 relates to reported operationalissues for the communication link 222 between the vehicle 220 and apedestrian 213.

Issue 221 concerns operational status of a drone. Such concerns mayreflect loss of or intermittent connections with ground-based equipment,overload, low power concerns, equipment malfunctions, etc. As notedabove, in some embodiments, a complete remote monitor station may beimplemented in drone.

This invention presents overall integrated artificial (AI) intelligencesystems and methods to analyze, prioritize and respond to all of theabove network type concerns with comprehensive network wide solutionsbased on relative issue criticality with AI decision making. Monitoredsensor signals may include audio, image, medical, process, material,manufacturing equipment, environmental, transportation, location,pipeline, power system, radiation, vehicle, computer, processor, datastorage, cloud processing, cloud data storage, drone, threat, mote, BOT,robot, telecommunication network or other followed sensor stationmonitoring signals.

FIG. 3 provides a more detailed exemplary configuration diagram for theremote sensor station 107 of FIG. 1—labeled remote sensor station 300 inFIG. 3. The remote sensor station 300 is controlled by processor 301which may be a microprocessor, computer or digital controller of thetype well known to those of skill in the art. The processor 301 isconnected to memory 317 for storage of programs and data for the remotesensor station 300. The processor 301 may include, without limitation,conventional processor or digital controller capabilities as well asmore specialized processing capabilities implemented for example withdigital signal processing (DSP). The memory 317 may comprise, withoutlimitation, solid-state random-access memory (RAM), solid-state readonly memory (ROM), and/or optical memory or mechanical, optical orsolid-state disk drive memory. The processor 301 also includes a powersupply 321 which may comprise, without limitation, batteries, AC/DCpower converters, solar cells, or other green energy power sources. Theprocessor 301 may also include input/output devices 320 comprisingwithout limitation USB ports, Common Flash memory Interface (CFI)approved by JEDEC and other standard memory unit interfaces. Theprocessor 301 also includes connection to a speaker 315 for broadcastingaudio signals from the remote sensor station unit 300.

Database access capability 318 of FIG. 3 may be implemented separatelyor as part of the system software operating on processor 301 for use inaccessing system parameters, control information, status information,history, audio recordings, video recordings, image recordings,operational information, contact information, internet addresses,telephone numbers, received messages, alarm signals and/or otherinformation used in the operation of the remote sensor station 300.

As further shown in FIG. 3, the processor 301 may include controls 316integrated with processor 301 for on/off control, controlling microphonesensitivity, speaker volumes, camera operations and/or other operationalfeatures of the sensor station 300.

As also shown in FIG. 3, the remote sensor station 300 may also includea directional microphone array 302 used to provide audio inputs from asubject being monitored as well as other audio signals from the areasurrounding the subject. The directional microphone array 302 may bedesigned to operate as a beamforming array with directional sound pickupforming a main lobe in the direction of the subject. The main lobe ismost sensitive to sounds emanating in the reduced area covered by thatlobe and is less sensitive to sounds emanating from other directions. Inthis way the directional microphone array 302 provides a noise orinterference reduction capability wherein the primary audio signalspicked up by the microphone array 302 are from the direction of thesubject being monitored. The directional microphone array 302 mayoperate with beamforming software executing on processor 301 asdiscussed more completely below. In addition to the directionalmicrophone array 302, but not illustrated in FIG. 3, single microphonesmay also be used in the present invention. Single microphones withpressure-gradient, directional acoustic sensitivity and/oromnidirectional microphones may be used as part of the remote sensorstation 300. Microphones based on variable capacitor technologies,electromagnetic induction technologies, fiber-optic technologies,variable resistance technologies, piezoelectric technologies andincluding MEMS (Micro-Electrical-Mechanical System) technologies may beused in the present invention.

The acoustical beamforming software may further comprise acoustical beamformer circuitry for receiving audio signals from each of multiplemicrophones comprising the directional microphone array, analog todigital conversion circuitry for converting the received audio signalsfrom each of the multiple microphones to digital signals, individualtransversal filters for filtering each individual digital microphonesignal, with the individual transversal filters further comprisingadaptive filter weights for each of the transversal filter tapped delayline signals, and with individual transversal filters further comprisingadditional circuitry for summing the outputs of the adaptively weightedtapped line signals producing an audio output signal most sensitive inthe direction of the main lobe of the sensitivity pattern of acousticalbeamforming circuitry. In this way, the acoustical beam-former signalsensitivity pattern may be adaptively varied with respect to directionof the main beam pattern lobe and null locations within that patternand/or sidelobe structure.

As further shown in FIG. 3, the remote sensor monitoring unit 300 mayalso include, without limitation, a background acoustic signal generator303. For example, a white noise generator may be used to generate whitenoise to be broadcast via speaker 315 depending on the subjectrequirements. The white noise generator is only meant to berepresentative of acoustic signal sources that may be used for thispurpose. Other possibilities include pink noise, announcements, audiomessages, soothing music generators or the like. In some embodiments itmay be desirable to avoid transmission of such background acousticsignals picked up by microphones 302 to monitor unit 102 of FIG. 1. Atthe same time, it may be desirable to ensure that the communication linksuch as link 204 of FIG. 2A is operational. As further explained below,this can be achieved by including known keep-alive signals in thetransmission to monitor unit 102 and suppressing the unwanted acousticsignal for transmission purposes to monitor unit 102 as described below.

As also shown in FIG. 3, the remote sensor monitoring unit 300 mayinclude an optical camera 304 which may be a video camera and/or acamera for capturing still images under control of the processor 301.The optical camera 304 may be used to monitor activities of the subjectas directed by processor 301 under program control.

In addition to the optical camera 304, the remote sensor monitoring unit300 may also include an infrared (IR) illuminator 305 operating undercontrol of the processor 301. The IR illuminator 305 may be used tobathe the subject and surrounding area with infrared illuminationincluding the possible use of structured infrared light. Such structuredlight may include, for example, multiple individual dots of infraredlight, vertical and/or horizontal raster scan infrared light orcirculars scans of infrared light. Infrared light is not visible to thehuman eye and has the advantage that it may be used in light or darkenvironments to create images or determine motions, movements oractivities of the subject being monitored.

In addition to the IR illuminator 305, the remote sensor monitoring unit300 may include an infrared IR camera 306 to detect reflected IRillumination from the subject and other objects in the illuminatedfield. The IR camera 306 may be used to generate image information foranalysis by the artificial intelligence image analysis capability 314discussed below. The goal of such image analysis may include thedetection of the dangerous situations or circumstances encountered bythe object or subject being monitored.

As also depicted in FIG. 3, the remote sensor monitoring unit 300 mayinclude environmental sensors 307 to further monitor and detectdangerous situations that may present themselves in the area of theobject or subject being monitored. Such sensors may be used to detect,for example, unacceptable temperature ranges, humidity, air pollutants,dangerous gases, fires or smoke.

The various sensors including but not limited to microphones, opticalcameras, infrared cameras, infrared sensors and environmental sensors ofFIG. 3 may all produce signals for transmission to the remote sensorstation 107 and/or the monitor unit 102 of FIG. 1. Various embodimentsof the remote sensor station 300 may implement radio RF transceivers forsuch communications using without limitation one or more of thetransceiver implementations illustrated in FIG. 3. Possible suchtransceivers include, without limitation, one or more RF antennas andtransceivers 312 designed for video, audio and/or data transmissionand/or reception. Transceiver 312 may use, for example, VHF, UHF,Cellular Telephone, Bluetooth or Wi-Fi or other signal transmissionspectrums. Modulation formats may include amplitude modulation (AM),single sideband (SSB) modulation, frequency modulation (FM), phasemodulation (PM) or other modulation schemes known to those of skill inthe art. Radio transceivers with appropriate antennas may also include aBluetooth transceiver 310, a Wi-Fi transceiver 311 and/or the cellulartelephone transceiver 309.

The remote sensor monitoring unit 300 Bluetooth, Wi-Fi, cellular orother RF transceivers of FIG. 3 may be used to send and/or receivesignals to/from the sensors 114 of the object or subject 101 asillustrated in FIG. 1. Such remote sensors may include audio, image,environmental, medical, process, equipment or other sensors to furtherdetermine and capture information concerning the object or subject 101of FIG. 1. The medical sensors 308 FIG. 3 may be implemented as part ofthe sensors 114 of FIG. 1 with RF or physical wire, cable or fiber opticconnections between the sensors 114 and the remote sensor station 107FIG. 1.

Medical sensors 308 implemented as part of the subject 101 sensors 114may include sensors for subject temperature, blood pressure, pulse rateor other sensors designed to monitor critical medical parameters for thesubject 101 of FIG. 1.

The remote sensor monitoring unit 300 of FIG. 3 may also include,without limitation, a GPS (Global Positioning System) 313 for anaccurate determination of the object or subject monitoring position.Other positioning technology not shown may include determining positionsbased on cellular telephone tower triangulation, Wi-Fi signal locationtechnology or other location determination capabilities known to thoseof skill in the art. The precise location of the remote sensor station300 may be important, for example, when reporting alarm conditions tothe monitor unit 102 or monitor center 115 of FIG. 1. Including suchlocation determination capability permits the remote sensor station 300to be moved from location to location or taken on trips and used forexample at homes, hotels, warehouses, factories, businesses or in othersuch locations or accommodations.

The remote sensor monitoring unit 300 of FIG. 3 may also include atime/clock unit 319 for providing accurate time information. Thetime/clock unit 319 may operate off the power supply unit 321 or the useseparate batteries or a power supply to ensure accurate timeinformation. Accurate time information may be used for example tocontrol operations of the object or subject sensor station microphones,acoustic signal generators, cameras, environmental sensors, and radiotransceivers discussed above. For example, it may be desirable toinitiate or terminate operation of the acoustic signal generators,cameras and/or other sensors under program control at specific times asdetermined by control parameter settings. It may also be important totransmit accurate time and/or date information with alarm signalstransmitted to the monitor unit 102 and/or monitor center 115 of FIG. 1.

Timing signals broadcast using AM, FM, shortwave radio, Internet NetworkTime Protocol servers as well as atomic clocks in satellite navigationsystems may be used in the present invention. For example, WWV is thecall sign of the United States National Institute of Standards andTechnology's (NIST) HF (“shortwave”) radio station in Fort Collins,Colo. WWV continuously transmits official U.S. Government frequency andtime signals on 2.5, 5, 10, 15 and 20 MHz. These carrier frequencies andtime signals are controlled by local atomic clocks traceable to NIST'sprimary standard in Boulder, Colo. These and other available timetransfer methods may be used in the subject monitoring system and methodof this invention.

Also depicted in FIG. 3 is artificial intelligence capability 314incorporating advanced signal processing for analysis of the varioussensors signals, evaluation of varying conditions and situationsconcerning the observed subject with integration of the results of suchobservation, analysis and evaluation into comprehensible usable outputsfor system control and/or alerting users of the monitoring system ofthis invention. Such artificial intelligence capability 314 may include,without limitation, image analysis, noise reduction, speech recognition,text-to-speech and speech-to-text conversion, natural languageprocessing, expert system analysis, fuzzy logic analysis and/or neuralnetwork analysis as discussed in more detail below.

For example, as discussed further below, image analysis may be used todetect changes in the object field being viewed, recognition ofparticular objects, facial recognition, recognition of movements orchanges in configuration the object or subject being viewed and thelike. Noise reduction may include time and/or frequency domain analysisof received video and/or acoustic signals to remove or reduce theeffects of background signal interference and noise to improve signalanalysis results. Speech recognition may be used to recognize particularspoken words, commands or phrases. Text-to-speech and/or speech-to-textconversion may be used to convert between textual and spoken or auditorymessages including commands, alerts and/or informative communicationsconcerning objects being monitored by the sensor monitoring network ofthis invention. Natural language processing may be used for automaticcontextual understanding of input signals or messages to facilitateautomatic response to those signals or messages.

FIG. 4 provides a more detailed exemplary configuration diagram for thesensor network monitor unit 102 of FIG. 1—labeled sensor network monitorunit 400 in FIG. 4. The sensor network monitor unit 400 is controlled byprocessor 401 which may be a computer, processor, microprocessor ordigital controller of the types well known to those of skill in the art.The processor 401 is connected to memory 412 for storage of programs anddata for the remote subject monitoring station 400. The processor 401may include without limitation conventional processor capabilities aswell as more specialized processing capabilities implemented for examplewith digital signal processing (DSP) or other well-known computertechnology. The memory 412 may comprise without limitation solid-staterandom-access memory (RAM), solid-state read only memory (ROM), and/oroptical memory or mechanical, optical or solid-state disk drive memory.The processor 401 also includes a power supply 416 which may comprisewithout limitation batteries, AC/DC power converters, solar cells, orother green energy power sources. The processor 401 may also includeinput/output device capabilities 415 comprising without limitation USBports, Common Flash memory Interface (CFI) approved by JEDEC and otherstandard unit interfaces. The processor 401 also includes connection toa speaker 411 for broadcasting audio signals from the sensor networkmonitor unit 400 or other sources as needed.

Also, like the remote sensor station 300 of FIG. 3, sensor networkmonitor unit 400 of FIG. 4 may also comprise artificial intelligencecapability 409. Artificial intelligence 409 may include for example andwithout limitation, image analysis, noise reduction, speech recognition,natural language processing, expert system analysis, fuzzy logic and/orneural networks. Including this capability in the remote subject sensorstation 300 of FIG. 3 may compliment that capability also contained inthe subject monitoring unit 400 of FIG. 4. In some embodiments of thisinvention such artificial intelligence capability may be implemented inonly one or both of the remote sensor station 300 and/or the sensornetwork monitor unit 400.

In some embodiments, as described above for FIG. 3 remote sensor station300, artificial intelligence 409 of the monitor unit 400 of FIG. 4 maycomprise image analysis used to detect changes in the object field beingviewed, recognition of particular objects, facial recognition,recognition of movements or changes in configuration the object orsubject being viewed and the like.

Artificial intelligence 409 of the monitor unit 400 of FIG. 4 mayfurther comprise noise reduction capability. In some embodiments,adaptive filtering may be implemented in the time or frequency domain.Such filtering is capable of isolating audible human speech signals frombackground white noise, pink noise or music signals and periodic signalssuch as the above described control tone frequencies. For example, anadaptive filter with frequency domain signal analysis may comprise FastFourier Transform (FFT) analysis to separate the received signal intofrequency subbands or bins, capability for analyzing said frequencysubbands or bins to isolate received noise signals from audio voicesignals in the frequency domain and eliminating noise signal levels inthe respective frequency subbands or bins. The adaptive filter may thenfurther combine the subbands or bins with reduced noise signal levelsand use Inverse Fast Fourier Transform (IFFT) analysis to generate timedomain signals representing the desired audio voice signals forbroadcasting via said remote subject monitoring station speaker. Theseand other techniques known to those skilled in the art may be used toseparate the desired subject audio signals from the background noise,interference or music picked up by the microphones 109 and broadcast tothe monitor unit 102 of FIG. 1.

Artificial intelligence 409 of the monitor unit 400 of FIG. 4 mayfurther comprise expert system analysis for derivation of controlsignals based on evaluation of received sensor signals with comparisonto expert control or output rules. Such expert system analysis is a formof artificial intelligence implemented to emulate the results of humanreasoning based on observed conditions. In some embodiments, fuzzylogic, an extension of expert system analysis, may be used whereinallowance is made for uncertainty in variable values with ranges ofvalues defined to accommodate such uncertainties. Ranges for particularvariables may in fact overlap depending on expert defined fuzzy logicrules or particular implementations. Expert systems output control ormessage information is derived to further emulate human reasoning basedon concerns, issues or uncertainties in observed conditions. In someembodiments, depending on the number of variables and expert systemrules defining variable relationships, the total number of suchrelationships may grow exponentially and complicate expert systemanalysis as explained further below. In some embodiments, hierarchicalexpert systems control may be used to offset such exponential growth incontrol and complexity as explained further below.

Artificial intelligence 409 of FIG. 4 may also comprise neural networks.Neural networks are yet another artificial intelligence tool for rapidevaluation of various combinations of inputs and derivation of outputs.Neural networks are based on networks of interconnected nodes emulatingthe structure and interconnection of neurons in the human brain. Suchneural networks are trained to respond in particular ways to particularcombinations of inputs.

Database access capability 413 of FIG. 4 may be implemented separatelyor as part of the system software operating on processor 401 for use inaccessing system parameters, control information, status information,history, audio recordings, video recordings, image recordings,operational information, contact information, internet addresses,telephone numbers, received messages, alarm signals and/or otherinformation used in the operation of the sensor network monitor unit400.

As also shown in FIG. 4 the processor 401 may operate with display 402for displaying images, control information or messages received by thesensor network monitor unit 400. Controls 403 are also integrated withprocessor 401 for on/off control, speaker volumes, transceiveroperations and/or other operational features of the sensor networkmonitor unit 400.

In addition, as also shown in FIG. 4, sensor network monitor unit 400may include RF antenna and receiver 404 compatible with the RF antennaand transceivers 309/312 of the remote sensor station 300 depicted inFIG. 3 and/or the monitor center 115 of FIG. 1. Similarly, sensornetwork monitor unit 400 may include, without limitation, a Bluetoothtransceiver 405, Wi-Fi transceiver 406 and/or cell phone transceiver 407for communication with subject monitoring unit 300 of FIG. 3 and/orsubject monitor center 115 of FIG. 1.

Also shown in FIG. 4 is noise and/or interference signal reductioncapability 408. As explained in more detail below, the present inventionincludes the capability to reduce or eliminate background interferenceand/or noise from signals received from the remote sensor station 300 ofFIG. 3. The remote sensor station 300 transmits audible sounds made bythe object or subject being monitored. In addition, as explained above,the remote sensor station 300 may also broadcast audible sounds such aswhite noise, pink noise or music into the environment of the object orsubject. As also explained above, the remote sensor station 300 mayinclude directional microphones or directional microphone arraysdesigned to primarily respond to audible sounds in the direction of thesubject. Nonetheless, additional background signals such as noise,music, etc. from the broadcast audible sounds may be picked up to somedegree by the remote sensor station 300 microphones 302. Other noise orinterference signals generated in the area occupied by the object orsubject may also be present in the signal transmitted to the sensornetwork monitor unit 400. Receiving these additional noise and/orinterference sources may be confusing, annoying and/or distracting tousers of the remote sensor station 400. It is the purpose of thekeep-alive signal processing 408 of FIG. 4 to substantially reduce oreliminate such extraneous noise from the broadcast by the speaker 411 ofFIG. 4 while at the same time not reducing or eliminating desiredaudible sounds generated by the subject or otherwise originating fromthe surroundings of the subject.

In one embodiment a frequency tone or digital control signal or thelike, referred to herein as “keep alive” signals, may be added to thenoise signal that is transmitted from the remote sensor station 107 ofFIG. 1. Such “keep alive” signals are added during periods of time whenno other audio signals other than the added background signals aredetected from the subject or subject surroundings being monitored. The“keep alive” signal is not broadcast by the remote sensor station 107into the area occupied by the subject. Rather it is added as a controlsignal to the signal transmitted from the remote sensor station 107 tothe monitor unit 102. Users of the monitor unit 102 may elect to: (1)always hear all signals transmitted from the remote sensor station 107to the monitor unit 102 including the added background noise or musicsignals, any transmitted control tone, and, of course, any audiblesignals from the object or subject being monitored; (2) to not hear theadded background signals but to always hear the transmitted keep-alivecontrol tone or signals, and again, of course, any audible signals fromthe subject or subject's surroundings being monitored; (3) only hearaudible signals from the subject being monitored.

Users may choose option (1) above when it is desired to hear all soundsfrom the area occupied by the object or subject. Users may choose option(2) or (3) above when it is desired to not hear the background noisesignals which can be very distracting or annoying, but at the same timeto receive a periodic keep-alive control tone for reassurance that thesubject monitoring system is actually operating correctly. When choosingoption (1) or (2), controls may adjust the time period between the “keepalive” signals to a time period acceptable to the user. Such time periodor interval adjustments may be made at the monitor unit 102 and/or theremote sensor station 107. The volume of “keep alive” control tonestransmitted via speaker 103 of FIG. 1 may also be adjusted.

Like the remote sensor station 300 of FIG. 3, the sensor network monitorunit 400 of FIG. 4 may also include a GPS receiver unit 410 for anaccurate determination of the monitoring unit position. Otherpositioning technology not shown may include determining positions basedon cellular telephone tower triangulation, Wi-Fi signal locationtechnology or other location determination capabilities known to thoseof skill in the art. The precise location of the sensor network monitorunit 400 may be important, for example, when reporting alarm conditionsto the monitor center 115 of FIG. 1. Including such locationdetermination capability permits the sensor network monitor unit 400 tobe moved from location to location or taken on trips and used forexample at hotels or in other accommodations.

In addition, like the remote sensor station 300 of FIG. 3, the sensornetwork monitor unit 400 of FIG. 4 may also include a time/clock unit414 for providing accurate time information as described above for theremote sensor station 300 of FIG. 3. The time/clock unit 414 may operateoff the power supply unit 416 or separate batteries or power supply toensure accurate time information. Accurate time information may be usedfor example to control operations of the sensor network monitor unit 400as discussed above. For example, it may be desirable to initiate orterminate operation of the keep-alive signal under program control atspecific times as determined by control parameter settings. It may alsobe important to transmit accurate time and/or date information withalarm signals and/or status signals transmitted to the network monitorcenter 118 of FIG. 1.

In addition, like the remote sensor station 300 of FIG. 3, the sensornetwork monitor unit 400 of FIG. 4 includes input/output ports 415 foraccess to the sensor network monitor unit 400. As is the case for FIG.3, these ports may include capabilities comprising, without limitation,USB ports, Common Flash memory Interface (CFI) approved by JEDEC andother standard unit interfaces.

FIG. 5 illustrates exemplary audio signals 500 of a type used in someembodiments in the subject monitoring system and method of thisinvention. Noise or other background signals 501 may be generated by themonitor unit 102 or the remote sensor station 107 of FIG. 1 andbroadcast by speaker 111 to be heard by subject 101. White noise is arandom signal with the uniform frequency spectrum over a wide range offrequencies. It is an electronically produced sound that may be somewhatsimilar to the sound of steady rain. In some embodiments pink noise 502may be used in place of the white noise 501. Pink noise is a randomsignal within the audible frequency range whose amplitude decreases asfrequency increases. In other embodiments different audible signals maybe used, for example, to calm the subject including different types ofnoise signals, music, a familiar person's voice or other audible signalspleasing to the subject 101.

Audio signal 503 is generated by subject 101 and may be speech signals,shouting or other audible signals. As can be seen from FIG. 5, thefrequency domain representation of audio signal 503 is quite differentfrom the background signals 501 or 502. Such differences in the time andfrequency domains permit noise reduction software to separate and/orreduce or eliminate audio noise signals 501 or 502 from the audio speechsignal 503.

It is to be understood that the microphone array 109 or other microphoneimplementations of the remote sensor station 107 of FIG. 1 may detectsounds in the area being monitored including sounds from signals such aswhite noise 501, pink noise 502, other noise sources such as music or asoothing voice, and other audio signals 503 from the subject 101.

In some embodiments, and as explained further below, the monitor unit102 may broadcast via speaker 103 the total received signal from theremote sensor station 107. In other cases, the user may elect to havethe remote sensor station suppress the noise or background component 501or 502 or the like in the received signal using noise or interferencereduction software as explained further below. However, the user maystill want to receive a “keep alive” or other control signal such as theperiodic sinusoidal signal 504 or the like for providing reassurance tothe user that the remote sensor station 107 is functioning properly. Inthis sense, the periodic sinusoidal or control signal may be considereda “keep alive” signal used to inform the user that the monitoring systemof the present invention is indeed operational even though no backgroundnoise or audible signals from the subject 101 are being heard.Suppressing the noise component of the received signal beforebroadcasting over the speaker 103 of the monitor unit 102 will result ina less confusing or annoying signal to be received by the user of thesubject monitoring system and method of this invention. Also, thecontrol capabilities 112 of the remote sensor station 107 will permitadjustment of the time between the audible sinusoid or other received“keep alive” signals. Such timing control signals may also betransmitted from the monitor unit 102 to the remote sensor station 107to select the time interval between “keep” alive signals.

The exemplary sinusoidal waveform 504 is depicted as comprising a sinewave signal of different or varying frequencies. In some embodiments, asingle sinusoidal or other predictable signal waveform may be usedinstead of the multiple frequency sinusoidal waveform 504. In otherembodiments, a digital control signal may be used in place of thesinusoidal signal 504 of FIG. 5. The periodic sinusoidal or othercontrol signal inserted into the transmitted signal from the remotesensor station 107 to the monitor unit 102 may be used in various ways.In addition to the keep alive signal, other control or messageinformation may be encoded into these signals.

FIG. 6 illustrates, without limitation, exemplary composite signals 600of the type described above. The exemplary composite signal 601comprises, for example, the white noise signal such as white noisesignal 501 of FIG. 5 with periodic insertion of the multi-frequencysinusoidal signal 504 of FIG. 5. FIG. 6 also depicts at 602 combiningthe pink noise signal 502 of FIG. 5 with a single frequency sinusoidalsignal as shown. As discussed above, when such signals are transmittedfrom the remote sensor station 107 to the monitor unit 102 of FIG. 1,the user may elect at the monitor unit 102 using controls 104 tosuppress the noise signals in 601, 602 or similarly constituted signals,while only hearing the sinusoidal or other “keep alive” signals. Inaddition, as discussed above, the periodicity of the audible sinusoidalsignals may be adjusted using the controls 112 of the remote sensorstation 107. In this way the user can eliminate the annoying audiblenoise signals while at the same time receiving periodic tones forreassurance that the subject monitor system and method are workingproperly. In yet another possible configuration, the user may elect tolisten to all signals including the noise signals and the periodicsinusoidal signals and subject audible signals and all other audiblesignals that may be detected by the microphones 109 of the remote sensorstation 107 of FIG. 1. In yet another configuration the user may electto only hear audible speech signals such as signals 503 of FIG. 5. It isclear from the above that the system and methods of the presentinvention provide maximum flexibility to the user in choosing whichaudible signals to hear while monitoring the subject 101.

FIG. 7 illustrates in more detail operations 700 involving themicrophone array 109 of the remote sensor station 107 shown in FIG. 1.The microphone array 109 is indicated as microphone array 703 in FIG. 7.The microphone array 703 is used in the implementation of thebeamforming and noise cancellation capability 700 of FIG. 7. Themicrophone array operates with beamforming software and/or hardware 704to form a directional acoustical beam 705 primarily sensitive to audiblesignals emanating in the area covered by such a directional beam 705.For example, configuring the beamforming capability to be directedtoward the subject 701 will result in primarily picking up audiblesignals in the direction of subject 701, while being less sensitive toother audible signals such as noise 702 illustrated in FIG. 7. In thisway audible signals that may mask desired audible sounds from thesubject 701 or otherwise confuse the user of the subject monitor system100 of FIG. 1 can be at least partially excluded from the detected audiosignal. In some embodiments, the beamforming noise cancellation systemmay be capable of automatically directing the beam to the audible soundsfrom the subject 701. In other embodiments, the beam may be manuallyadjusted or be directed by physical placement of the remote sensorstation 107 and the subject 701.

As further illustrated in FIG. 7, the beamforming noise cancellationcapability 700 may also include additional signal processing capability706. This signal processing capability may include further noisereduction for extraneous or leakage noise sources outside the main beamlobe. In addition, this further signal processing may be used to add thesinusoidal or other control “keep alive” signals 504 of FIG. 5 to thenoise signals to result in composite signals such as signals 601 and 602of FIG. 6. As explained above, these additional sinusoidal or othercontrol signals may be used to selectively eliminate background noisesignals transmitted from a speaker 103 of the monitor unit 102 of FIG. 1thereby reducing or eliminating background noise that the user may findannoying or confusing when monitoring the subject's activities.

As further illustrated in FIG. 7, the composite audio output signal maybe passed to transmitter 707 and antenna 708 for transmission to themonitor unit 102 and/or the monitor center 115 of FIG. 1. As describedabove, this transmitter may be implemented, without limitation, with aBluetooth, Wi-Fi, cellular telephone or other appropriate signaltransceiver for communications.

Also illustrated in FIG. 7 are control signals including beamformingcontrol 709, signal processing control 710 and transmitter control 711.These control signals may be used to select particular operations of theacoustic beamforming, signal processing and transceiver capabilities. Itis to be understood that all of the beam forming, signal processing andcontrol may be implemented separately or integrated with the processorcontrol of FIGS. 3 and/or 4.

FIG. 8 illustrates, without limitation, exemplary controls 800 for thesubject monitoring system and methods of the present invention. Thesecontrols may be implemented in a variety of ways familiar to those ofskill in the art. For example, dials and/or switches may be employed toselect specific control options. In some embodiments, control optionsmay be shown on the display 105 of monitor unit 102 as illustrated inFIG. 1 and/or other displays not shown on the remote sensor station 107.It is to be understood that, in different embodiments of this invention,the exemplary control options of FIG. 8 may be implemented in both themonitor unit 102 and the remote sensor station 107 or be implemented injust one of the monitor unit 102 and the remote sensor station 107, orthose control options may be distributed with some being implemented inthe monitor unit 102 while others are implemented in the remote sensorstation 107 of FIG. 1. That is to say, remote control capabilities maybe distributed between the monitor unit 102 and the remote sensorstation 107 using the communication capabilities of those units. In someembodiments, control options may also be executed by remote processorsincluding cloud processor and storage configurations. In someembodiments control processing may be implemented on a distributed basisbetween monitor unit 102, the remote sensor station 107 and the cloudcomputing such as cloud computing 120 in FIG. 1. Distributed processingmay also extend to the network monitor center 118 of FIG. 1. Results ofsuch distributed processing may be displayed on display 801 and/or otherdisplays associated with distributed processing configurations.

Exemplary display 801 is illustrated in FIG. 8 with indicated displayicons 802. The display 801 may be controlled from external controls suchas a keyboard or mouse pointer arrangement. In some embodiments, thedisplay 801 may also be implemented, without limitation, as aninteractive, touchscreen display with display options selected by thetouch of a finger and/or an appropriate stylus depending on theimplementation of the touchscreen display. Possible interactive displaytechnologies include, without limitation, capacitive touchscreens,electromagnetic radiation sensitive touchscreens, optical sensitivetouchscreens and pressure sensitive touchscreens. Touchscreens sensorsmay be implemented as an integral part of the display 801 with controlelectronics integrated into that display. Alternatively, sensors locatedaround the periphery of the display 801 may be used to determine the XYcoordinates or positions of a finger or stylus used to select particularicons.

FIG. 8 also provides, without limitation, exemplary control options forthe monitoring system and methods of the present invention. Nineexemplary high-level controls are indicated which may be accessed, forexample, through the nine indicated icons 802 of the display 801. Inaddition to control of the display 801, eight high-level icons permituser selection of the particular features to be controlled withexemplary features including, without limitation, audio signals, videosignals, medical signals, process signals, followed IOT sensor signals,transceivers, telecommunication network alerts and artificialintelligence settings. Selecting any one of these control options mayresult in the dynamic changing of the interactive control screen 801 todisplay another level of control options. An example of such operationis illustrated in FIG. 8 wherein selection of the top-level icon 803results in opening of three additional icons 804. Additional levels ofcontrol may also be implemented wherein selecting one of the icons 804will result in opening yet another level of control icons or objects.

For example, selecting the high-level “display” icon may open additionalicons for further control of the display including options, for exampleand without limitation, screen configuration, brightness and/or zoomcontrol. Selecting the high-level “display” icon may also, for example,open additional icons for control of the display of video signals usedto capture activities of the subject 101 and/or activities in the areasurrounding the subject 101 of FIG. 1.

Selecting the high-level audio signal icon may open additional levels ofcontrol for selecting, for example, white noise or pink noise asillustrated in FIG. 8 or other audible signals not shown. Additionalcontrol options may include time periods during which the backgroundwhite noise or pink noise or other suitable sounds are to be broadcastfrom the remote sensor station 107 to the area surrounding the subject101. Other control options may include keep alive signal parameters, thevolume of the background sounds and/or other sound control parameters.

As also illustrated in FIG. 8, selecting the high-level audio signalsicon may give access to further lower level icons to control acousticalbeamforming operating with a microphone array 109 of the remote sensorstation 107 of FIG. 1. Beamforming options may include, withoutlimitation, enabling acoustical beamforming, disabling acousticalbeamforming, beam configuration, beam width, and/or selectivelydirecting the main beam in selected directions depending upon thephysical configuration of the subject environment and the object orsubject 101 relative to the remote sensor station 107 of FIG. 1. Whileacoustic beamforming serves to reduce pickup of extraneous noise by themicrophone array, depending on the situation, leakage noise from othersources not located in the main lobe of the acoustic beam may also bepresent. For this reason, it may be desirable to implement yet furthernoise reduction prior to transmission to the monitor unit 102 of FIG. 1.Control options for this purpose may include enabling or disabling suchadditional noise reduction and/or selecting particular parameters to beused in further noise reduction algorithms.

Selecting the high-level “audio signal” icon in FIG. 8 may also provideuser control over the signal to be transmitted from the remote sensorstation 107 to the monitor unit 102 of FIG. 1 of the present invention.Several options for such signals may exist including those listed inFIG. 8. In one exemplary option, the transmitted signal includes thebackground noise or other accompanying sounds plus any subject audiosignals detected by the microphones 109 of FIG. 1. A second exemplaryoption transmits the same signals and also includes the periodic “keepalive” signal such as the “keep alive” sinusoidal signals depicted inFIGS. 5 and 6 and described above. As described above such “keep-alive”signals may be used by the remote sensor station 107 of FIG. 1 tocontrol the broadcast of selected signals via speaker 111 of the remotesensor station 107. As also discussed above, this capability permitssuppressing or eliminating confusing or unwanted background signals fromthose audible signals transmitted to the monitor unit 102 while stillensuring that the subject monitoring system is operating properly andwhile still permitting transmission of audible sounds from the object orsubject 101. Yet a third signal selection option indicated in FIG. 8permits transmission of only the keep alive sinusoidal signals of FIG. 6together with any object or subject 101 audible sounds. This optionsuppresses the background noise signals from the signal actuallytransmitted from the remote sensor station 107 to the remote monitorunit 102 of FIG. 1. In a yet another option only detected audible soundsfrom the object or subject 101 may be broadcast by the remote sensorstation 107 speaker.

Video signal controls indicated in FIG. 8 may include controls forselecting full-motion video, still images or the use of infraredimaging. Particular image analysis software may also be selected for thecorresponding image capture.

As further indicated in FIG. 8, in some embodiments one of control iconsthe control display 801 may be dedicated to the analysis of medicalsensors for persons being monitored. Such sensors may include, forexample and without limitation, temperature sensors, blood pressuresensors, cardiac sensors and/or oxygen sensors. Medical sensors may, forexample and without limitation, be worn by the persons being monitored,implemented in the person's clothing or implanted in the persons beingmonitored.

As also indicated in FIG. 8, in some embodiments a control icon forprocess monitoring may be implemented in the display 801. Such processesmay include, for example and without limitation, manufacturingprocesses, material flow processes and logistic control processesinvolving movement materials or products or the transportation needs forsuch movement.

In some embodiments it may be desirable or even important to be madeaware monitoring and control results for other remote sensor stations asindicated in the sensor monitoring network of FIG. 2A. In an aspect ofthis invention such monitoring of the remote sensor stations may beimplemented by designating certain other remote sensor stations to be“followed” by an individual remote sensor station. As indicated in FIG.8, such following may include audio/video/medical/process alerts fromother remote sensor stations. Other shared alerts may include, withoutlimitation, alerts for weather, traffic, crowds, pipeline status,utility power systems, emergency alerts and even terrorist alerts. Inthis way a given remote sensor station may be made aware of situationsof concern at other remote sensor stations. These other remote sensorstations may be in close proximity to the following remote sensorstation. In addition, such “following” may be used to track results fromother remote sensor stations not necessarily in close proximity to theremote sensor station but still important in the evaluation of overallsituations of concern.

Selecting the high-level signal transceiver icon of FIG. 8 provides usercontrol of selected radio frequency or other transmission systemcapabilities for communications between the monitor unit 102, the remotesensor station 107 and/or the monitor center 118 as illustrated inFIG. 1. As indicated in FIG. 8, selection of the signal transceiver iconmay cause opening of additional icons on the interactive displaysallowing, without limitation, configuration of system communications foruse of Bluetooth, Wi-Fi or cellular technology. For example, withoutlimitation, such communications with the monitor center 118 may beconnected through cellular telephone, microwave, fiber-optic link, cablecommunications, wired connections or other appropriatetelecommunications media. Also, the monitor unit 102, may communicatethrough Wi-Fi or Bluetooth connections to a local router for connectionto broader telecommunication networks. In some cases, it may beappropriate to also specify data rates, transmission times, transmissionformats, or other telecommunication system parameters for operativeconnection to the chosen telecommunications media.

Selecting the high-level artificial intelligence (AI) icon permitsmanaging the use of the various AI options including parameterselections, signal processing operations, and control and/or warningsignal generation as discussed above and in greater detail below.

In the various embodiments discussed above, selecting a high-level iconin FIG. 8 enables user selection and enablement of particular alarmconditions for the generation of alarm signals. Selecting particularicons may open additional icon selections for configuration ofparticular alarm situations. As shown in FIG. 8, possibilities include,without limitation, audio alarms, video alarms, medical alarms, processalarms, followed TOT alarms, telecommunication network alarms. Forexample, alarm conditions for monitoring subject people may includemovements, extended silence, medical conditions, presence of anintruder, undesired presence of pets or animals, and unacceptableenvironmental conditions such as out of range temperature or thepresence of dangerous gases, smoke or fire. Medical alert conditions mayinclude, without limitation, lack of response from the subject, subjecttemperature, blood pressure, oxygen levels or pulse rate. In someembodiments, the subject may wear a medical monitoring device not shownsuch as an arm or leg bracelet to monitor critical subject medicalparameters.

As also discussed above, transmitted alarm signals may also includespecific location information such as, without limitation, GPS locationof the object or subject 101. Including this capability permits themonitor unit 102 and/or the remote sensor station 107 to be moved fromplace to place or even carried on a trip to a distant location whilestill being operative to transmit alarm signals to/from the monitorcenter 118 or other appropriate locations with those alarm signalsincluding the current location of the object or subject situationneeding attention. Knowing the location can be important in derivationof information or warning messages depending on the situations atparticular locations.

As further described below, FIGS. 9, 10A and 10B illustrate, withoutlimitation, an exemplary flowchart for operations for the exemplarynetworks of FIGS. 1, 2A and 2B as discussed above. In some embodimentsthe operations of FIGS. 9, 10A and 10B may be distributed between theremote sensor station 107, the monitor unit 102 the network monitorcenter 118 or other accessible distributed cloud or processingcapabilities of FIG. 1. The operation may be automatically or manuallyinitiated at the start block 901. The control setup and initiate block902 analyzes and initializes the various control inputs from the usersof the subject monitoring system of this invention. Such control inputsmay include, without limitation, for example, those control operationsdescribed in FIG. 8 above.

Having initiated the subject monitor unit 102, control is passed to oneor more of exemplary sensor input blocks audio sensor inputs 903, imagesensor inputs 904, medical sensor inputs 905, process sensor inputs 906,followed remote sensor station inputs 907 and/or telecommunicationnetwork sensor inputs 908. It is to be understood that some embodimentsmay have a subset comprising one or more but not necessarily all ofthese sensor inputs 903, 904, 905, 906, 907 and 908. Sensor inputs ofFIG. 9 may include detection of network cyberattacks and malicioushacking of network connected devices and sensors, telecommunicationfacilities, data processing and storage facilities, and informationcollected, stored, transmitted and processed in the IoT network. Also,other sensor inputs not specifically included in FIG. 9 may be used insome embodiments without departing from the teachings of thisdisclosure.

Exemplary more detailed identification of possible capabilities of eachof the sensor inputs 903-908 are provided via flowchart connectors A-Fidentified as 909, 910, 911, 912, 913 and 914 respectfully in FIGS. 9,10A and 10B. Based on analysis of the identified sensor inputs, asdescribed in more detail below, exemplary expert systems warning indicesare derived at 915 categorizing each as presenting very low, low,medium, high or very high danger or concern for each of the monitoredsubjects or objects as discussed above. Based on these categorizations,artificial intelligence expert systems or fuzzy logic analysis 916provides comprehensive composite derivation of appropriate warningsignals informing users of where the most urgent problems may exist anddirecting corrective actions according to the relative degrees of dangerfor each monitored subject or object as described above. Exemplaryartificial intelligence analyses are described via connector G 917 ofFIG. 9 at FIGS. 12-19 as indicated.

Based on the above artificial intelligence analysis, warning alarmdecisions 918 are made. If no alarms are indicated, control is returnedto control block 902 via path 920. If alarm signals indicate the needfor corrective actions, those alarms are transmitted at 919 and controlreturned to control block 902 via path 920.

Analysis of exemplary audio sensor inputs 903, image sensor inputs 904and medical sensor inputs 905 via connectors A 909, B 910 and C 911 ofFIG. 9 are shown in more detail in analysis diagram 1000 of FIG. 10A.The connectors A 1001, B 1007 and C 1011 correspond to connectors A 909,B 910 and C 911 respectfully of FIG. 9. Similarly, analysis of exemplaryprocess sensor inputs 906, followed IOT sensor inputs 907 andtelecommunication network sensor inputs 908 via connectors D 912, E 913and F 914 of FIG. 9 are shown in more detail in the continuation ofanalysis diagram 1000 of FIG. 10B. The connectors D 1016, E 1018 and F1020 correspond to connectors D 912, E 913 and F 914 respectfully ofFIG. 9. It is to be understood that the analysis capabilities depictedin FIGS. 10A and 10B are exemplary. Other embodiments may have a subsetof the illustrated capabilities of FIGS. 10A and 10B or othercapabilities not shown without departing from the teachings of thisdisclosure.

Referring now to Background Audio Processing 1002, as explained aboveand illustrated in FIGS. 5 and 6, the audio sensor signal analysis mayinclude background audio processing with broadcast audible sounds suchas white noise, pink noise, music or other selected sounds into theenvironment of the object or subject being monitored. These backgroundsounds may be picked up by sensor microphones and transmitted back tothe sensor stations. In some embodiments or operations, it may bedesirable to suppress these known background sound signals prior tolistening to or recording sounds from the monitored subject or object.The Background Audio Processing 1002 may provide such audio signalfiltering to remove these retransmitted background signals. As alsoexplained above and illustrated in FIGS. 5 and 6, when such knownbackground signals are removed, it may be desirable to transmit periodic“keep alive” signals from the subject or object monitors to ensure thatthe units are operating correctly despite such selective backgroundsignal suppression. For example, this capability may be especiallybeneficial when monitoring infants or other persons in need of specialcare.

As explained above and illustrated in FIG. 7, Acoustic Beamforming 1003may be implemented in some embodiments with a microphone arraycomprising one or more microphones used to listen to sounds from theobject or subject being monitored. Arrays permit the use of microphonebeamforming in the direction of sounds emanating from the subject whilereducing sounds received from other directions. Microphone arrays usebeamforming software to create acoustical beams with a main lobedirected at the object or subject for attenuating or perhaps essentiallyeliminating sounds arriving at the microphone from different directions.In some embodiments, acoustic beamforming 1003 makes use ofanalog-to-digital converters with transversal filters for filteringindividual digital microphone signals with adaptive filter weightspermitting adaptive variation of the acoustic beam pattern and sidelocal structure.

As also explained above, in some embodiments acoustic noise reduction1004 may comprise time and/or frequency domain noise reduction adaptivefiltering. Including the use of Fourier transform analysis with thefrequency domain divided into sub-bands for frequency selective signalevaluation.

As also explained above, in some embodiments speech recognition 1005 maybe used to recognize particular spoken words, commands or phrases.Natural language processing 1006 may be used for automatic contextualunderstanding of input signals or messages to facilitate automaticresponse to those signals or messages.

As also explained above, image sensors 904 of FIG. 9 may comprise, forexample, video, still image, or infrared visual monitoring of the objector subject 101 and/or the surrounding area. Exemplary image analysiscapability is depicted in FIG. 10B via connector 910 of FIG. 9 andcorresponding connector 1007 of FIG. 10A.

Analysis of image sensor signals may include image filtering/enhancement1008 in the time and/or frequency domain depending on the application.Time domain filtering may be used to recognize time varying situationsin the received image sensor signals such as movements of the object orsubject being monitored or new image features. Frequency domainfiltering including the use of Fourier transforms or Fast Fouriertransforms (FFT) may be used to analyze and capture frequency domaincharacteristics of individual images including gradient changes in imageintensity or contrast that may be compared from image to image to assistin ascertaining changes in the image content. Two-dimensional frequencydomain filtering of portions or all of an image may be used. Time domainand/or frequency domain filtering may be used to improve image qualityfor further analysis.

Image pattern recognition and/or feature extraction 1009 may be used todiscover particular patterns or features in the captured images. Suchpattern recognition and/or frequency extraction may include capabilitiessuch as facial recognition or recognition of particular patternsexpected to be found in the image and alerts when such patterns are notfound.

Image/Tile feature comparisons 1010 may be used for individual images,successive images or frames of video images to monitor for changes inthe captured image which may indicate concerns about image content. Insome embodiments images may be segmented into multiple tilesrepresenting a subset of a total image with comparison of tile contentfrom image to image to monitor for changes in tile image. Such changesmay indicate potential problems or concerns about the subject or objectbeing monitored.

As also explained above, medical sensors 905 of FIG. 9 may be analyzedvia connector 911 represented as connector C 1011 in FIG. 10A. Suchmedical sensor analysis may comprise temperature sensor analysis 1012,blood pressure sensor analysis 1013, pulse sensor analysis 1014 oroxygen sensor analysis 1015 or other medical sensors used to monitor themedical condition of a subject. Other medical sensors may includesensors implanted in the body or tissues of a subject being used fortracking selective medical parameters or bodily functions of thesubject.

Sensor signal analysis 1000 of FIG. 10A is continued in FIG. 10B.Analysis of process sensor inputs 906 of FIG. 9 are provided byconnector D 912 at corresponding connector D 1016 of FIG. 10B listingexemplary process sensor analysis operations 1017. As indicated at 1017process sensor analysis may include inputs from process equipmentsensors. Such process equipment may include, without limitation,manufacturing equipment, robotic equipment, BOTS, production linemonitoring equipment, transportation equipment including air, ground andrail transportation, chemical processing equipment, drones, computerprocessing equipment, data storage equipment, or any equipment used tofacilitate process operations.

As also indicated in the process sensor analysis 1017, input sensor dataindicating the status of process materials necessary for the executionof particular processes may be collected for analysis. Such data mayreflect process material parameters such as material availability,quantity, quality, alternatives, location or other parametersdescriptive the materials necessary to carry out a given process.

As also indicated in the process sensor analysis 1017, input sensor dataindicating process maintenance requirements including maintenanceschedules, required downtimes, maintenance results, and projectedmaintenance issues based on process equipment status monitoring. Suchstatus monitoring may indicate, for example and without limitation,operational parameters indicative of future performance issues that mayarise in the execution of the process.

As also indicated in the process sensor analysis 1017, input sensor dataindicating process schedules including precedent requirements forexecution of particular processes and materials required for thatexecution. Such scheduling may impact other operations throughout theInternet of Things being monitored. Informing monitoring units incontrollers throughout the network may greatly facilitate efficientexecution of distributed operations in that network.

As further indicated in FIG. 10B, followed IOT sensor monitor unitinputs 908 from FIG. 9 may be analyzed via connector E 914 of FIG. 9represented by corresponding connector F 1018 of FIG. 10B. In someembodiments, it may be important for individual Remote Sensor Stations107 and/or Monitor Units 102 of FIG. 1 to be made aware of the status ofresults from other Remote Sensor Stations or Monitor Units locatedthroughout the network. In this case, individual Remote Sensor Stationsor Monitor Units may elect to “follow” or be informed of the monitoringresults of those other Remote Sensor Stations or Monitor Units. Factorsthat may enter into such decisions to “follow” other stations or unitsmay include, without limitation, proximity to those other stations orunits, criticality of operation of those other stations or units,concerns for dangerous conditions being detected at those other stationsor units and/or dependence on the operation or status of other objectsor sensors being monitored at those other units. In this way, theoverall operations of the present IOT monitoring system may be extendedto include distributed interdependent or critical operations beingmonitored and analyzed throughout the network.

As indicated at 1019, parameters and/or analysis results from other“followed” remote sensor stations or monitor units may include resultsfrom audio sensors, image sensors, medical sensors, process sensorsand/or telecommunication network sensors as discussed above. Reportingfrom such “followed” stations and units may include an evaluation of theoverall stations or unit monitoring signal status and indication of anydangerous or other concerns arising from such an evaluation.

As further indicated at 1019, multiple other potential situationsincluding environmental concerns such as whether, critical traffic orcrowd concerns, critical equipment failures such as failures involvingpipelines or public utilities or police reported criminal activitiesincluding terrorist alerts may be widely reported throughout the networkto all remote sensor stations and monitoring units that have chosen tofollow such concerns. In some cases, such concerns may be widelyreported to remote sensor stations or monitoring units without thosestations or units registering to receive warnings for such concerns.

As further indicated in FIG. 10B, telecommunication network sensorinputs 908 from FIG. 9 may be passed to connector F 914 to correspondingconnector F 1020 of FIG. 10B. Exemplary sensor networks communicationelements including selected telecommunication elements and potentialtelecommunication network faults are depicted in FIGS. 2A and 2B asdiscussed above. As shown in FIG. 10 B 1021 telecommunication networksensors may include individual link sensors including transmitters,receivers, repeaters, antennas, signal amplifiers, link power sources,physical wireline or optical cables and other resources involved inimplementing a particular telecommunications link. In addition, asindicated at 1021, telecommunication sensors may provide signal qualitysignals for individual links including measurements such as signalpower, signal-to-noise ratio, multipath interference issues, adjacentchannel interference, data error rates or other parameters indicative ofthe signal quality on respective telecommunication links.

As also indicated in FIG. 10B, individual link modems may be monitoredwith modem sensors providing information on operational parameters forthe modems for various modulation formats including AM, FM, PM, signalmultiplexing formats and multiple other parameters. Problems or issuesinvolving the signal modulation can be detected and reported for furtherevaluation.

In addition, as indicated at 1021, signal link quality and/or modemsensor monitor parameters and switching node parameters may also bemonitored using sensors to indicate the operational capability of suchswitching nodes including failures and traffic congestion.

Modern communication networks make use of data routers to routeindividual data packets to desired destinations. Such routers are usedat both the network level and at the periphery the network includingcommonly used Wi-Fi routers and routers in cellular networks. Monitoringof such routers as also indicated that 1021 of FIG. 10B.

As also indicated at 1021 of FIG. 10B and illustrated at 219 in FIG. 2B,the monitoring systems of this network may include detection of failuresof telecommunication subnetworks. Such a failure will clearly affect allsensor nodes in the failed subnetwork. In addition, failures of aparticular subnetwork may impact IoT sensors and monitoring nodes notdirectly connected to that failed subnetwork. For example, a particularsubnetwork may correspond to resources necessary to rectify problems inother areas of a total network. A global failure of that subnetworkcould impact objects or situations located in other parts of the totalnetwork. In another example, failures in a subnetwork may correspond topower outages in particular areas. Such power outages may impactadvisable activities in other areas of the total network. In yet anotherexample, failures corresponding to release of dangerous gases, liquidsor other materials in one subnetwork may impact other areas of the totalnetwork. Failure of the subnetwork communications may interruptcommunications with other portions of total network to report thedangerous situation.

In some embodiments, the various sensor measurements discussed above mayvary with time. Time series analysis may be used to maintain historyfiles with analytic evaluation of those files to determine parameterranges, changes in parameter values, statistical analysis of successiveparameter values, regression analysis, predictions, or other analyticevaluation of the time varying sensor inputs. Without limitations, timevarying sensor measurements may be collected in cloud storage facilitieswith history files and access to big data analytics provided for theanalysis of such data files. Based on such analysis, machine learningbased on statistical trends in parameter values may lead to improvedperformance.

It is to be further understood that while sensor signal analyses of FIG.10A and 10B are shown as serial operations, such signal analysisoperations may be carried out selectively, in parallel, or on adistributed basis without departing from the teachings of thisinvention. It is also to be understood that other and/or additionalsensor signal analysis operations not specifically set forth in FIGS. 9,10A and 10B above may be employed without departing from the teachingsof this invention.

FIG. 11 depicts, without limitation, an exemplary expert mapping 1100 ofsensor signal analysis results to an exemplary expert sensor ratingscale 1101. Example rankings 1102 include derived parameters for audiosensor signals, video sensor signals, medical sensor signals, processsensor signals, followed IoT remote sensor monitor unit sensor signals,and telecommunication network signals. It is to be understood that othersensor parameters may be used in addition to or in place of theexemplary parameters of FIG. 11 without departing from the teachings ofthis invention. Derived signal parameters are mapped, for example, ontoa scale from 0 to 1. Mappings of 0 correspond to derived parameters thatpresent absolutely no danger or concern to the object or situation beingmonitored. Mappings of 1 correspond to derived parameters that indicatethe maximum danger or concern to the object or situation beingmonitored. Mappings may be defined by experts. As further explainedbelow and illustrated on the exemplary sensor ranking scale 1101 of FIG.11, the degrees of danger between 0 and 1 may be partitioned intodefined ranges corresponding to very low danger, low danger, mediumdanger, high danger, or very high danger. Such partitioning may beuseful in simplifying reporting results or in implementing artificialintelligence expert system analysis and/or fuzzy logic analysis and thederivation of overall danger warning and/or control signals.

Returning now to the exemplary listing of parameters at 1102 of FIG. 11,audio speech signals may be parsed for selected keywords being ranked ona scale from 0 to 1. Particular ranking may depend upon the subjectbeing monitored. A crying screaming infant may be ranked higher on thescale than other audible signals from the infant such as harmlessbabbling. At the same time, and infant continually asking for “mommy”may be ranked higher than harmless babble but lower than crying andscreaming. Spoken words from older persons may be ranked according tothe urgency that such words they convey. Words or phrases such asrobbery, gun, don't shoot, help, I have fallen, heart attack, danger orgun shots would receive a higher ranking than phrases or wordsexpressing less urgency. In addition to speech recognition for keywords,natural language processing may be used to derive complete phrases orsentences from the detected audio signals. Modern natural languageprocessing ascertains meaning of such complete phrases or sentencesuseful in understanding concerns or dangerous situations being describedby a speaker. Here again, expert defined rankings from 0 to 1 on thescale 1101 may be made.

In a similar manner results from image, video or infrared signalprocessing 1007 may be ranked on a scale from 0 to 1 as shown in FIG.11. Changes in the image fields may indicate a range of dangeroussituations or situations of concern. Any movement in a field-of-viewthat is supposed to be stationary may be a concern. Image analysis mayinclude definitions of smaller sections of image defined as tiles withcomparison of tiles from one image to the next for changes. Motions inthe image field being monitored outside the area were the subject islocated may indicate dangerous situations. Detection of rapid changingmovements of individuals in the field-of-view may indicate a conflict oreven fighting. Facial recognition may also be used to generateappropriate warning signals on the defined scale of FIG. 11.

As further indicated in FIG. 11, medical sensor signal analysis 1011 maybe used to monitor parameters such as the subject temperature, bloodpressure, pulse, oxygen levels or other important medical parameters.Here again the individual parameters may be ranked on a scale from 0 to1 depending on their deviation of expected normal readings. Medicalsensors may be attached to the subject, implanted in a subject,integrated into the subject's clothing or otherwise worn or placed indefined proximity to the subject for monitoring purposes.

As further indicated in FIG. 11, process sensor signal analysis 1017 maybe used to monitor the state of a defined process. An example would bemonitoring of a manufacturing process with sensors used to monitormanufacturing equipment, materials used in the manufacturing process,situations requiring maintenance, automated robotic equipment and/ormonitoring of the entire process with comparisons to requiredmanufacturing schedules. Other processes that may be monitored include,for example, chemical processes, transportation processes, computerprocesses including data processing and storage systems and interrelatedactivities of personnel in the execution of particular processes.

In some embodiments, it may be desirable for a given remote sensorstation 107 in FIG. 1 to communicate or follow results from otherdifferent remote sensor stations as indicated at 1019. The results fromthose other different remote sensor stations may indicate a dangeroussituation or situations of concern for the given remote sensor station.For example, an emergency detected at a different remote sensor stationin defined proximity to a first remote sensor station may serve as analert calling for action at that first remote sensor station. Also, forexample, any of the above audio, image, medical or process sensorparameters at the remote sensor station may give rise to such an alert.Other situations detected at other remote sensor stations that may because for concern include, but are not necessarily limited to, badweather, traffic congestion, crowds, gas or water or petroleum orchemical pipeline failures, utility systems including electricutilities, police emergencies or terrorist alerts and the like. In somecases, these alerts from other remote sensor stations may be morecritical when the remote sensor stations are the close proximity to oneanother.

As also indicated at 1102 of FIG. 11, telecommunication network sensorsignal analysis may include multiple communication parameters asdiscussed above. FIG. 11 identifies, without limitation, multiple suchparameters including communication links error rates, signal-to-noiseratios, traffic congestion delays, lack of telecommunication systemresponse, reported link outages, reported processing node outages,reported storage outages and reported subnetwork failures as discussedabove. Here again, expert input may be used to map suchtelecommunication issues onto the expert sensor ranking scale 1101.

FIG. 12A depicts in matrix form artificial intelligence expert systemrelationships 1200 between two selected parameters resulting fromanalysis of audio sensor inputs and video sensor inputs that may be usedin some embodiments of the present invention. As indicated in FIG. 12A,the ranges for audio and video parameters are divided into exemplarysubranges corresponding to very low, low, medium, high and very high asshown in FIG. 11. For each combination of such values for the twoparameters being considered in FIG. 12A, an artificial intelligenceexpert decision rating is provided indicating degree of danger indicesfor each commination. These danger indicators are provided by audio andvideo parameter analysis experts and are part of an artificialintelligence expert system database. As indicated in FIG. 12A, thecombined audio/video danger indicators may be defined by such expertsas, for example, being very low, low, medium, high, and very high. Inthe exemplary embodiment depicted in FIG. 12A, twenty-five such expertsystem defined rules are shown.

The combined warning/control matrix 1200 of FIG. 12A forms the basis ofan artificial intelligence intelligent system. For example, each of theresults indicated in FIG. 12A may be expressed in propositional calculuslogic form, for example, as follows:

1. If audio danger is low and the video danger is high, then thecombined warning and control index is high.

2. If audio danger is very high and the video danger very low, then thecombined warning index is very high.

3. If audio danger is very low and video danger is medium then thecombined warning and control index entry is medium.

Clearly 25 such logical statements exist for the entries in FIG. 12A.For each such logical statement, a combined warning and control indexfor the given combination may be determined by the expert system of thepresent invention. The combined warning and control index may bedisplayed on the display 801 of FIG. 8 in various forms including textmessages, flashing, with audible messages from the speakers of FIGS. 3and/or 4 or with a combination of such visual or audible alerts.

The artificial intelligence expert system matrix of FIG. 12A issymmetric with respect to the two variables, audio and video. That is tosay, every row of the matrix is identical to the corresponding columnwith the result that the matrix entries are symmetric about each of thetwo diagonals. The result is that equal weight is given to each of thetwo audio and video variables.

It is also to be understood that in some embodiments it may be desirableto establish priorities between sensor variables. For example, it may beimportant to prioritize certain medical sensor parameters over audiosensor and over sensor parameters. One way of favoring one variable overanother is to add an expert defined value to the result obtained fromthe expert sensor ranking scale of FIG. 11. For example, if thecalculations for ranking a variable resulted in a value of 0.3 in FIG.11, adding 0.1 to that value would increase the danger warning for thatparticular variable from “low” to “medium.” The subsequent expert systemor fuzzy logic analysis would then favor the results derived from themodified sensor values sensors over the results from unmodified sensors.

In still other embodiments, where it is desirable to prioritize or givemore weight to a selected parameter, an unbalanced expert defined matrixmay be used. One such exemplary unbalanced matrix is shown in FIG. 12B.For example, in the case where the audio signal analysis indicates avery low danger and a video signal analysis indicates a high danger,then the combined warning and control index is set at medium asindicated in FIG. 12B. In this case, the very low audio warning andcontrol index is given priority or increased weight. Even though thevideo warning and control index is high, the artificial intelligenceexpert system analysis results in lower danger warning and control indexof medium. If both audio and image parameters are high, the combinationis an output warning index of very high, indicating that the highcombination of two inputs is interpreted as a very high warningcondition. The matrix of 12B is defined by an expert and reflects theexpert's opinion concerning importance of individual variables andcombinations of those variables.

In other embodiments of the present invention, the above describeddecision-making process may be augmented with the use of fuzzy logic. Itis clear from the above discussion that the audio and video parametervalues will be variables with certain ranges of uncertainty. Forexample, in the analysis of FIG. 11, hard boundaries are set between thedifferent ranges of very low, low, medium, high and very high. Thesehard boundaries do not actually exist in the real world. Human decisionmaking is more nuanced and not subject to such binary decisions based ondefined limits or boundaries. In some embodiments, analyses that providefor a more gradual transition between defined ranges are moreappropriate. As described below, artificial intelligence expert systemsusing of fuzzy logic is particularly well-suited for deriving controlrules for variables with such uncertainty. It is to be understood thatartificial intelligence expert system derivations may be implementedwithout fuzzy logic as described above. The use of the above describedexpert defined propositional logic rules may be sufficient for someembodiments as described above. That said, fuzzy logic has foundexpanded uses in the development of sophisticated control systems. Withthis technology, complex requirements may be implemented in relativelysimple, easily managed and inexpensive controllers. It is a simplifiedmethod of representing analog processes on a digital computer. It hasbeen successfully applied in a myriad of applications such as flightcontrol systems, camera systems, antilock brakes systems, washingmachines, elevator controllers, hot-water heaters, decision analysis,and stock trading programs.

With fuzzy logic control, statements are written in the form of thepropositional logic statements as illustrated above. These statementsrepresent somewhat imprecise ideas reflecting the states of thevariables. The variable ranges for audio and video parameters indicatedmay be “fuzzified” as fuzzy logic variables extending over the definedoverlapping ranges as shown, for example, in FIG. 13 at 1300. Fuzzylogic systems make use of “fuzzifers” that convert input variables intotheir fuzzy representations. “Defuzzifiers” convert the output of thefuzzy logic process into “crisp” numerical values that may be used insystem control. It is to be understood that while the exemplary fuzzylogic analysis of FIG. 13 is based on the illustrated triangularmembership functions, other such overlapping membership functions may beused such as Gaussian, exponential or other functions selected forparticular applications.

For example, the graph 1301 of FIG. 13 illustrates such a possible“fuzzification” for the audio warning and control index variable withoverlapping ranges indicated in the figure. In this example, on a scaleof 0 to 1.0, the normalized audio warning and control index from FIG. 11is rated at 0.85. As illustrated in the FIG. 13, an audio warning andcontrol index rating of 0.85 results in a degree of membership (DOM) of0.70 in the membership class “high.” In this particular example, thewarning and control index rating of 0.85 does not result in membershipin any other of the possible membership classes.

In a similar way, in the graph 1302 of FIG. 13 “fuzzification” of thevideo warning and control index variable is illustrated. On a scale of 0to 1.0, a normalized video warning and control index value of 0.45results in a DOM of 0.6 in the video “medium” membership class and 0.15in the “low” membership class. These DOM values may in turn be used inthe fuzzy logic implementation to derive a defined, “crisp” numericalvalue for a combined warning and control index.

In the above example of FIG. 13, the composite warning and control indexdepends on the degrees of membership of the audio signal analysis “or”the video signal analysis. The conjunctive relation “or” corresponds tothe logical intersection of the two sets corresponding to the audio andvideo variables. In this case the appropriate DOM is the maximum DOM foreach of the sets at the specified time. This is expressed algebraicallyas follows:

(A∪B)(x)=max(A(x),B(x)) for all x∈X

Premises connected by an “AND” relationship are combined by taking theminimum DOM for the intersection values. This is expressed algebraicallyas follows:

(A∩B)(x)=min(A(x),B(x)) for all x∈X

The conjunctive relation “or” requires the use of the maximum value ofthe respective DOM's. These values may be used to defuzzify the warningand control index degree of membership. As shown in 1303 of FIG. 13,fuzzy ranges for the warning and control index may be defined in asimilar manner to the audio and video variables. A numerical “crisp”value for the warning and control index can now be derived usingdefuzzification procedures. As shown in FIG. 13, the DOM ranges for thewarning and control index are capped at values corresponding to theabove analysis for the DOMs of the audio and video variables. The final“crisp” numerical value of the warning and control index may, forexample, be calculated based on the centroid of the geometric figure forthe DOM ranges of the graph 1303 of FIG. 13. This calculation may becarried out by dividing the geometric figure of FIG. 13 into sub-areasA_(i), with individual centroids x_(i) from the following formula.

$x_{c} = {\left( {\sum\limits_{i = 1}^{n}\;{x_{i}\mspace{14mu} A_{i}}} \right)\text{/}\left( {\sum\limits_{i = 1}^{n}\; A_{i}} \right)}$

The result of such a calculation is shown in FIG. 13 yielding a warningand control index numerical value of about 0.6. Note that this result isless than the warning and control index for audio signals alone and morethan result for video signals alone. Fuzzy Logic produces a result inbetween these extremes, reflecting the fuzzy transitions from onedesignated range to another.

While, for simplicity, the above example dealt with only two variables,the audio signal analysis results and the video signal analysis results,the method described above may be expanded to more than two variables,including multi-dimensional expert system analysis and multi-dimensionalfuzzy logic analysis. Multi-dimensional fuzzy logic may be applied tothe example parameter combinations of FIGS. 14-17 discussed below.

For example, while the above example is limited to two variables, audioand video, clearly, for some embodiments, additional tables may beconstructed to include other important variables in the decisionprocess. Multidimensional tables may be constructed with more than twovariables to reflect additional indices. Exemplary other parameters mayinclude, the results of analysis for medical, process, followed IoTsensor monitor units and telecommunication network analysis as describedabove. For the case of two variables, audio and video with 5 definedsubranges of variable values, 25 possible combinations exist. As thenumber of variables increases, the number of possible combinationsincreases exponentially. For example, with 6 such variables and 5 rangesof values for such variables, the number of possible combinationsincreases to 5⁶=15,625 possible outcomes. Modern processing and memorysystems make artificial intelligence expert systems analysis of such alarge volume of possibilities manageable. In one embodiment of thisinvention, this approach with, perhaps a reduced number of ruleseliminated, a subset of all possible rules that may not apply to a givenanalysis is used.

However, clearly a simpler artificial intelligence expert systemimplementation is desirable. In another embodiment of this invention, ahierarchical artificial expert and/or fuzzy logic system is disclosedthat reduces the increased size of the inference rule data base withaddition of more variables from exponential growth to linear growth.Hierarchical fuzzy system designs are discussed, for example, in the G.Raju, L. Wang and D. Wang references cited above in the identificationof prior art in this patent. In addition, the hierarchical systems andmethods of this invention implement MIMO (Multiple input-MultipleOutput) operations with intermediate evaluation of dangerous situationspermitting response to such situations in addition to providingevaluation of the levels of concern or dangerous situations for thecombination of considered variables. In some embodiments, adaptivefeedback control is provided to further improve hierarchical systemcontrol and processing of input signals.

An exemplary hierarchical MIMO adaptive operation 1400 is illustrated inFIG. 14 for a system with 4 input variables 1401: X1, X2, X3 and X4. Inthis case, the maximum number of required AI expert system rules isreduced from 5⁴=625 to 3×5²=75. More generally, for a hierarchicaldesign with “n” input variables and “m” possible values per variable,required AI expert system rules will be (n−1)m². Thus, the number ofrequired rules is a linear function of the number of variables asopposed to the exponential increase m^(n) in the non-hierarchical case.As another example, for a system with 6 input variable and 5 subsetranges of values for each variable, the number of required AI expertsystem rules is reduced from 5⁶=15,625 to 5×5²=125. The result is asignificant decrease in design and implementation complexity of thesystem. The advantage of such a reduction in a system with a largenumber of variables and variable ranges is clear. In some embodiments ofthis invention with many network wide variables and variable ranges asubstantial reduction in complexity can be achieved.

In FIG. 14, the inputs 1401 are processed at block 1402 using, forexample, input signal processing 1403 and signal ranking methods of FIG.11. Signal processing may include signal filtering, noise reduction,analog to digital conversion, audio signal processing, speechrecognition, natural language processing, video signal processing, imageanalysis, signal time series analysis and statistical signal analysis asdiscussed above. Analysis of additional signals from other followedremote sensor station and/or monitor units and/or network monitorcenters and network telecommunication failures as indicated in FIGS. 1and 2 and discussed above may also be included in the signal evaluationsof blocks 1402 and 1403 of FIG. 14. The individual processed inputs 1401may be mapped onto a scale from 0 to 1.0 with zero being input signalindicative of no danger or concern for an individual parameter and 1.0being indication of maximum danger or concern for that parameter. Asindicated in FIG. 11 and at 1402 of FIG. 14 the individual inputparameters X1, X2, X3 and X4 may also be mapped into the ranges very low(VL), low (L), medium (M), high (H), or very high (VH) according totheir numerical rankings representing the degree of danger or concernpresented by the parameter values. The resulting output processedsignals 1404 from blocks 1402 and 1403 are designated X1 , X2 , X3 andX4 in FIG. 14.

As shown in the embodiment of FIG. 14, outputs X1 , X2 , X3 and X4designated 1404 from the signal processing and adaptive ranking ofblocks 1402 and 1403 are mapped to signals Y1, Y2, Y3 and Y4 for furtherprocessing by the hierarchical adaptive expert system controller. Theoutputs of the hierarchical controller may be considered anapproximation to an output derived using a complete non-hierarchicalexpert system controller as described above. In some embodiments of thisinvention, the results obtained with the hierarchical controller may beimproved with selective control of routing of input signal tohierarchical control levels. For example, the specific routing of thesignals X1 , X2 , X3 and X4 to the signals Y1, Y2, Y3 and Y4 may beadaptively changed depending on signal input importance, outputsensitivity to particular input signals or other parameterrelationships. (See, for example, Di Wang, et. al., and F. Chung citedabove.) In some embodiments, it may be desirable to adaptively applyinputs with more specific information first, and inputs with lessspecific information later in the hierarchical network of FIG. 14. Forexample, input signals with the largest values may be deemed moreimportant than input variables with smaller values. In otherembodiments, particular input parameters may be considered moreimportant than other input variable. For example, in some embodiments,medical signals may be deemed more important than other signals. Also,the input signal of most importance may change with time. For example,time series analysis of particular signal inputs may indicate trends ofconcern for particular inputs. The output parameters Y1 and Y2 arepassed to AI expert system analysis block 1406 for AI expert systemanalysis to determine the degree of danger or concern Z1 represented bythe particular combination of Y1 and Y2.

With the hierarchical analysis of some embodiments of this invention,the output parameter Z1 is also passed to AI expert system analysis 1407together with the output parameter Y3 for an analysis and determinationof the degree or danger or concern Z2 represented by this combination.In this way, the output of AI expert system analysis 1406 is used as aninput to AI expert system analysis 1407 in a hierarchical manner forsuccessive computations.

In the same hierarchical manner, the output Z2 of AI expert systemanalysis 1407 is passed input to AI expert system analysis 1408 togetherwith the output Y4 from the input processing and ranking block 1402 fordetermination of the degree of danger or concern Z3 represented by thecombination of the variable Y4 and the output Z2.

The input variables X1, X2, X3 and X4 are represented collectively by1401 in FIG. 14. The output variables Y1, Y2, Y3 and Y4 are representedcollectively by 1405. The results Z1, Z2 and Z3 of the AI expert systemanalyses 1406, 1407, and 1408 are represented collectively by 1409 inFIG. 14. The combination of variables 1405 and 1409 are fed to block1410 for further signal analysis and issuance of appropriatewarning/control signals. In this way, the warning/control signals may beindicative of individual concerns or danger arising from the outputvariables 1404 as well as the results of the AI expert system analyses1406, 1407 and 1408 as illustrated in FIG. 14. The output Z3 includesresults based on all four input variables 1401. In addition, in someembodiments, further feed-back control signals 1411 may be used toprovide further control of selection of signal ranking 1404 dependingsensitivity or other measurements of output signals to selection orordering of signals 1404 for analysis as indicated in FIG. 14. Thehierarchical system and method of FIG. 14 is a MIMO (MultipleInput-Multiple Output) hierarchical AI Expert system with 4 inputs and 7outputs.

Returning now to the two-input audio/video system of FIGS. 12A, 12B and13, in another embodiment of this invention, a third variable may beadded to the audio/video (AV) analysis using the above outlinedhierarchical MIMO AI adaptive expert system analysis of FIG. 14. Forexample, FIG. 15 adds the medical variable resulting in an AVM analysis.In this example, a non-symmetric expert system matrix 1500 as discussedabove is used. Depending on the expert input, other matrices may alsomay be more appropriate. As discussed above, non-symmetric matricespermit favoring certain variable or variable combinations over others.For example, in the matrix of FIG. 15, the medical parameter is givenmore weight than the audio or video parameters. The result of the expertsystem analysis of FIG. 15 is an AVM (audio/video/medical) combinationwarning and control index output. The variable relationships of FIG. 15may also be processed using fuzzy logic as shown in FIG. 13.

In the same way, the derived process signal value may be added to thehierarchical adaptive expert system analysis as shown in the matrix 1600of FIG. 16. The result of this analysis is a warning and/or control(AVMP) combination signal output warning and control index foraudio/video/medical/process as indicated in FIG. 16. The samemodifications for emphasizing certain variables over others and fuzzylogic formulations as described above can applied to the AVMPcombination calculation. Here again, the variable relationships of FIG.16 may also be processed using fuzzy logic as shown in FIG. 13.

In the same way, the derived “followed” remote sensor station signalvalue may be added to the hierarchical adaptive expert system analysisas shown in the matrix 1700 of FIG. 17. The result of this analysis is awarning and/or control (AVMPF) signal output foraudio/video/medical/process/followed combination as indicated in FIG.17. The same modifications for emphasizing certain variables over othersand fuzzy logic formulations as described above can applied to the AVMPFcalculation.

In the same way, derived “telecommunication network” remote sensorstation signal values may be added to the hierarchical expert systemanalysis as shown in the matrix 1800 of FIG. 18. The result of thisanalysis is a warning and/or control (AVMPFT) signal output warning andcontrol index for audio/video/medical/process/followed/telecommunicationcombination as indicated in FIG. 18. The same modifications foremphasizing certain variables over others and fuzzy logic formulationsas described above can applied to the AVMPFT calculation.

The above described operations may involve the input of multipleexemplary parameters such as the audio, medical, process, followedremote sensor stations, and telecommunication sensor signals, and mayin-turn result in the output of a single composite warning and controlsignal based on the combination the input sensor signals. Such systemsare sometimes referred to as MISO systems with multiple inputs and asingle output. Systems with multiple inputs and multiple outputs aresometimes referred to as MIMO systems. In some embodiments of thepresent invention, MIMO operation permits generation of multiple controloutputs based on multiple sensor signals inputs as described above.

FIG. 19 depicts one such possible hierarchical MIMO embodiment 1900 ofthe present invention. In this embodiment, multiple outputs may begenerated based on multiple calculated results indicating therequirements for urgent responses to multiple sensor signal inputs. Theflowchart of FIG. 19 provides for multiple intermediate outputsdepending on the results that may develop in the processing, analyzingand evaluating the input data.

Here again, the flowchart of FIG. 19 is a continuation of the flowchartof FIG. 9 via the connector 1921 G. At block 1901 audio/video (AV)sensor signals are input for evaluation. Block 1902 computes an AVwarning and control index as outlined for example in FIGS. 12A, 12B and13. Control is then passed to the urgent decision block 1903. At thispoint the decision is made as to whether the audio signals alone, videosignals alone or the computed AV itself require an issue of a warning orsystem control signal. If a warning signal is to be issued at this pointcontrol is passed to the issue warning block 1904 for generation anddispatch of that signal. Once that warning is issued or in the eventthat no urgent warning signal is necessary, control is passed to theinput medical sensor network data block 1905.

The received medical sensor data is in turn passed to block 1906 forcomputation of the audio/video/medical (AVM) warning and control indexusing, for example, the expert system matrix of FIG. 15 or fuzzy logiccalculations as indicated in FIG. 9 at block 916. At this point thedecision is made as to whether the medical signals alone or the computedAVM itself require an issue of a warning or system control signal.Having computed the AVM warning and control index at block 1906, controlis passed to the urgent decision block 1907. If a warning or controlsignal is to be issued at this point control is passed to the issuewarning block 1908 for generation and dispatch of that signal. Once thatwarning is issued or in the event that no urgent warning signal isnecessary, control is passed to the input process sensor network datablock 1909.

The received process sensor data is in turn passed to block 1910 forcomputation of the audio/video/medical/process (AVMP) warning andcontrol index using, for example, the expert system matrix of FIG. 16 orfuzzy logic calculations as indicated in FIG. 9 at block 916. At thispoint the decision is made as to whether the process signals alone orthe computed AVMP itself require an issue of a warning or system controlsignal. Having computed the AVMP warning and control at block 1910,control is passed to the urgent decision block 1911. If a warning signalis to be issued at this point, control is passed to the issue warningblock 1912 for generation and dispatch of that signal. Once that warningis issued or in the event that no urgent warning signal is necessary,control is passed to the input followed sensor network block 1913 fordata inputs from other remote sensor stations of interest as describedabove.

The received “followed” sensor data is in turn passed to block 1914 forcomputation of the audio/video/medical/process/followed (AVMPF) warningand control index using, for example, the expert system matrix of FIG.17 or fuzzy logic calculations as indicated in FIG. 9 at block 916. Atthis point the decision is made as to whether the “followed” signalsalone or the computed AVMPF itself require an issue of a warning orsystem control signal. Having computed the AVMPF warning and controlindex at block 1914, control is passed to the urgent decision block1915. If a warning signal is to be issued at this point, control ispassed to the issue warning block 1916 for generation and dispatch ofthat signal. Once that warning is issued or in the event that no urgentwarning signal is necessary, control is passed to the inputtelecommunication sensor network data block 1917 for data inputs fromtelecommunication network components and/or telecommunicationsubnetworks as discussed above.

The received telecommunication network sensor data is in turn passed toblock 1918 for computation of theaudio/video/medical/process/followed/telecommunication (AVMPFT) warningand control index using, for example, the expert system matrix of FIG.18 or fuzzy logic calculations as indicated in FIG. 9 at block 916. Atthis point the decision is made as to whether the telecommunicationsignals alone or the computed AVMPFT itself require an issue of awarning or system control signal. Having computed the AVMPFT warning andcontrol index at block 1918, control is passed to the urgent decisionblock 1919. If a warning signal is to be issued at this point, controlis passed to the issue warning block 1920 for generation and dispatch ofthat signal. Once that warning is issued or in the event that no urgentwarning signal is necessary, control is passed back to block 916 OF FIG.9.

It is clear from the above discussion of the flow chart of FIG. 19 thatmultiple warning signals may be issued based on the analysis outlined inthe flowchart. With multiple input signals and multiple output signalsthe disclosed system operates as a hierarchical multi-variable MIMOsensor network warning and/or control system.

FIG. 20 is an exemplary artificial neural network of the type useful insome embodiments of this invention. Artificial neural networks are inpart modeled after operations in the biological neural network of thehuman brain. It has been estimated that the biological neural network ofthe human brain contains roughly 100 billion neurons. Biological neuralnetwork neurons interact and communicate with one another viainterconnecting axons and dendrites. Biological neurons respond to aweighted combination of input signals with comparison of the sum toactivation thresholds. A single biological brain neuron may receivethousands of input signals at once that undergo the summation process todetermine if the message gets passed along to other neurons in thebiological network.

Artificial neural networks as shown in FIG. 20 are based on a simplisticmodel compared to the actual biological neural network of the brain.Nonetheless, artificial neural networks are proving useful, for example,in problems encountered in pattern recognition, facial recognition,prediction, process modeling and analysis, medical diagnostics anddynamic load scheduling. Referring to FIG. 20, the artificial neuralnetwork 2000 receives inputs 2006 from external sensors. The nodes 2004of the artificial neural network 2000 correspond approximately toneurons in the human brain and are organized with an input layer 2001and output layer 2003 interconnected by hidden layers 2002. The nodesare interconnected by neural network connections 2005. Weighted sums ofinput signals may be compared to threshold levels within the neuralnetwork nodes to produce output signals for transmission to subsequentnodes or outputs 2007. For example, in image recognition problems theinputs 2006 correspond to signals from image sensors. The output signals2007 will indicate whether or not the image being observed correspondsto a desired image. The internal weights and summing operations areconfigured during a training process corresponding to the actual desiredresult. Multiple training methodologies have been proposed including theuse of backward chaining feedback arrangements of the output signals toadjust the weights of the artificial neural network summing operationsto achieve the proper result if the desired image is presented.Artificial neural networks may thus be characterized as containingadaptive weights along paths between neurons that can be tuned by alearning algorithm from observed data using optimization techniques inorder to improve the model.

(See, for example,https://www.innoarchitech.com/artificial-intelligence-deep-learning-neural-networks-explained!)

FIG. 21 depicts, without limitation, an exemplary sensor network neuralnetwork expert system analysis 2100 in accordance with the systems andmethods of the present invention. Audio, image, medical, process,material, manufacturing equipment, environmental, transportation,location, pipeline, power system, radiation, vehicle, computer,processor, data storage, cloud processing, cloud data storage, drone,threat, mote, BOT, robot, telecommunication network or other followedremote sensor station monitoring signals comprise the sensor networkinputs 2101. In some instances, the signal inputs are processed byartificial neural networks 2102 as illustrated in FIG. 21. The outputsfrom the neural networks along with the additional sensor signal inputsare processed by the sensor signal analysis/neural networkanalysis/ranking operations 2103. The processing 2103 accesses expertsystem rules 2104 for derivation system warning and control signals asdiscussed above. In some embodiments, fuzzy logic inference rules mayalso be accessed to provide fuzzy logic warning and control signals asdiscussed above. In some embodiments, hierarchical adaptive MIMO controlmay be implemented as described above.

FIG. 22 illustrates in more detail exemplary fuzzy logic systemoperation execution 2200 useful in the system and methods of thisinvention. As shown in FIG. 22, the operations of fuzzy logic inferenceengine 2201 include access to the artificial intelligence expert systemknowledge base 2205 which may include the fuzzy logic rules discussedabove. The fuzzy logic operations include the fuzzifier 2202 used toestablish degree of memberships DOMs as discussed above. The outputs offuzzifier 2202 are fed to the fuzzy logic processing element 2203.Defuzzifier 2204 provides crisp numerical outputs for the warning andcontrol index 2206 as discussed above.

Systems and methods described above provide network wide IoT warning andcontrol signals. MIMO embodiment operation including input analysis fromsensors directly connected to a given remote sensor station as well asinputs from different remote sensor stations designated as beingfollowed by a given sensor station are disclosed. Embodiments based onartificial intelligence expert systems analysis, fuzzy logic analysisand the use of neural networks are included in systems and methodsset-forth in this invention. Expert system and fuzzy logicimplementations with hierarchical control and/or adaptive feedback aredisclosed.

It should be understood that the drawings and detailed descriptions arenot intended to limit the invention to the particular form disclosed,but on the contrary, the intention is to cover all modifications,equivalents and alternatives falling within the spirit and scope of thepresent invention as defined by the appended claims.

1. An artificial intelligence IoT (Internet of Things) Big Datainformation management and control system comprising IoT networkperformance analysis, fault detection and expert system generation ofwarning and control signals indicative of selected, critical IoT elementperformance issues or faults further comprising the followingcombination of capabilities: expert predesignated data collection sensorstations comprising one or more dedicated sensors directly connected toor contained within specifically programmed computer system executableprogram code expert predesignated data collection sensor stations; afirst expert predesignated data collection sensor station furthercomprising: an artificial intelligence processing warning and controlsystem with at least one specifically programmed computer system furthercomprising expert system processing of captured IoT sensor datainformation with interpretation of that information based on humanexpert defined parameters, human expert defined parameter ranges, humanexpert defined parameter range subsets, and human expert definedpropositional logic rules based on parameter values; a memory forstoring said captured IoT sensor data information and further forstoring said specifically programmed computer system executable programcode; at least one processor for executing said specifically programmedcomputer system executable program code; at least one transceiverconnection for communicating via IoT communication links; said IoTcommunication links providing said first expert predesignated datacollection sensor station connections to receive information signalsindicative of operational status of selected expert predesignated datacollection sensor stations other than said first expert predesignateddata collection sensor station or selected expert predesignatedtelecommunication equipment, all selected with a first expertpredesignated data collection sensor station centralized controller forperformance or fault analysis based on criticality to said first expertpredesignated data collection sensor station operations; said firstexpert predesignated data collection sensor station receiving multipleinput signals comprising: (1) input signals from one or more dedicatedsensors directly connected to or contained within said first expertpredesignated data collection sensor; (2) input signals from one or moreother expert predesignated IoT data collection sensor stations that aredifferent from said first expert predesignated data collection sensorstation wherein said input signals are indicative of operational statusof said different expert predesignated IoT data collection sensorstations; and (3) input signals indicative of the operational status ofexpert predesignated IoT distributed telecommunication equipment; saidfirst expert predesignated data collection sensor station furthercomprising input signal processing by said centralized controller andadaptive ranking by said centralized controller of importance of saidinput signals (1), (2), and (3) with signal parameter evaluation andassignment to expert defined ranges depending on level of concernindicated by said signal parameter evaluation for use in expert systemanalysis; said first expert predesignated data collection sensor stationartificial intelligence processing further comprising multi-stagehierarchical expert system analysis by said centralized controller ofsaid individual processed and ranked input signals (1), (2), and (3) andgeneration by said centralized controller of: (4) each successiveindividual output from each stage of said multi-stage hierarchicalexpert system analysis including the final stage output; generation ofwarning and control signals based on each of individual processed andranked input signals (1), (2), and (3) and further based on the outputsignals (4) of each stage of said multi-stage hierarchical expert systemanalysis including the final output stage; routing each of saidindividual processed and ranked input signals (1), (2), and (3) to bothseparate output signal processing and also to successive stages of saidmulti-stage hierarchical expert system, and routing the output of eachsuccessive individual output from each stage of said hierarchical expertsystem analysis except the final stage output to both said output signalprocessing and to a subsequent stage of said multi-stage hierarchicalexpert system, and also routing hierarchical expert system analysisfinal stage output to the separate output signal processing; generatingindividual warning and control signals based on individual networkelement performance or failure issues and also based on multi-stagehierarchical expert system analysis by said centralized controller ofcombinations of multiples of those individual network elementperformance or failures issues; signal processing receiving each ofseparate individual processed and ranked input signals (1), (2), and (3)in addition to (4) each successive individual output from each stage ofsaid hierarchical expert system analysis including the final outputstage for use in derivation of adaptive feedback control by saidcentralized controller for use in dynamic ranking of input signalspresented to said expert system analysis; said artificial intelligenceprocessing further comprising, in addition to expert system processing,one or more of neural network processing, image processing, noisereduction processing, speech recognition processing or fuzzy logicprocessing; and whereby said combination of capabilities provides bothIoT Big Data performance analysis and fault detection efficiency byminimizing Big Data access and processing operations achieved throughexpert predesignation of critical IoT network elements withcommunication of information indicative of status of said critical IoTnetwork elements to said first expert predesignated data collectionsensor station, and further with information processing based onartificial intelligence, efficient multi-stage hierarchical expertsystem signal processing with fewer propositional logic statements thanused in non-hierarchical expert system signal processing, and furthercomprising adaptive feedback control, and distributed warning andcontrol signal generation and processing.
 2. The artificial intelligenceIoT (Internet of Things) Big Data information management and controlsystem of claim 1 wherein said expert defined propositional logic rulescomprise defined signal input ranges for each input signal representedin a multidimensional matrix of conditional output matrix entries, andfurther wherein said conditional output matrix entries are based oninput signals from separate controllers from at least two of thefollowing sources: (a) selected first expert predesignated datacollection sensor station, (b) selected expert predesignated datacollection sensor stations other than said first expert predesignateddata collection sensor station, or (c) selected expert predesignatedtelecommunication equipment.
 3. The first expert predesignated datacollection sensor station of claim 2 wherein said input signals mayinclude audio input signals, image input signals, medical input signals,process input signals, material input signals, manufacturing equipmentinput signals, environmental input signals, transportation inputsignals, location input signals, pipeline input signals, power systeminput signals, radiation detection input signals, vehicle input signals,computer input signals, processor input signals, data storage inputsignals, cloud processing input signals, cloud data storage inputsignals, drone input signals, threat detection input signals, mote inputsignals, BOT input signals, robot, telecommunication network inputsignals, cyberattack input signals, malicious hacking input signals orother expert predesignated data collection sensor station input signals.4. The first expert predesignated data collection sensor station ofclaim 2 wherein said expert defined propositional logic rules are basedon priorities or importance of selected object or situation expertdefined parameters or on selected combinations of object or situationparameters.
 5. The first expert predesignated data collection sensorstation of claim 2 wherein said artificial intelligence processingfurther comprises neural network processing with backward chaining fromcomputed results to improve future computational results.
 6. Theartificial intelligence IoT (Internet of Things) Big Data informationmanagement and control system of claim 2 wherein said first expertpredesignated data collection sensor station further comprises access ofsaid first expert predesignated data collection sensor station tointernet cloud storage and processing units.
 7. The artificialintelligence IoT (Internet of Things) Big Data information managementand control system of claim 2 wherein said first expert predesignateddata collection sensor station further comprises sensor inputs that mayvary with time with time series analysis of time variable sensor inputdata.
 8. The first expert predesignated data collection sensor stationof claim 8 wherein said time series analysis includes regressionanalysis of time varying sensor signal parameter values.
 9. Theartificial intelligence IoT (Internet of Things) Big Data informationmanagement and control system of claim 2 wherein said first expertpredesignated data collection sensor station communicates with othernetwork nodes to track connected telecommunication network elements,subnetworks or networks for failures or performance issues impactingsaid first expert predesignated data collection sensor station.
 10. Theartificial intelligence IoT (Internet of Things) Big Data informationmanagement and control system of claim 2 wherein said first expertpredesignated data collection sensor station further communicates with aterrestrial or air-born vehicle or is implemented in a terrestrial orair-born vehicle.
 11. The artificial intelligence IoT (Internet ofThings) Big Data information management and control system of claim 2wherein said first expert predesignated data collection sensor stationfurther communicates with drone or is implemented in a drone.
 12. Theartificial intelligence IoT (Internet of Things) Big Data informationmanagement and control system of claim 2 wherein said first expertpredesignated data collection sensor station further communicates with arobot or is implemented in a robot.
 13. The artificial intelligence IoT(Internet of Things) Big Data information management and control systemof claim 2 wherein said first expert predesignated data collectionsensor station further communicates with a BOT or is implemented in aBOT.
 14. The artificial intelligence IoT (Internet of Things) Big Datainformation management and control system of claim 2 wherein said firstexpert predesignated data collection sensor station further communicateswith a mote or is implemented in a mote.
 15. The artificial intelligenceIoT (Internet of Things) Big Data information management and controlsystem of claim 1 wherein said expert predesignated data collectionsensor stations are further connected to a monitor unit, said monitoringunit providing collection, processing and consolidation of informationfrom one or more expert predesignated data collection sensor stationsand communication with other network nodes.
 16. The artificialintelligence IoT (Internet of Things) Big Data information managementand control system of claim 15 wherein said monitored Internet of Things(IoT) objects or situations comprise one or more persons and furtherwherein said person may be an infant, child, invalid, medical patient,elderly or special needs person.
 17. The artificial intelligence IoT(Internet of Things) Big Data information management and control systemof claim 16 wherein said first expert predesignated data collectionsensor station broadcasts background audio signals in the area of saidperson.
 18. The artificial intelligence IoT (Internet of Things) BigData information management and control system of claim 17 wherein saidfirst sensor network remote sensor station broadcasted background audiosignals are removed from or attenuated in signals transmitted to aconnected, separate monitor unit to minimize annoying or unnecessarysignals received and/or heard at said separate monitoring unit whilestill transmitting audio signals from a monitored object or person. 19.The artificial intelligence IoT (Internet of Things) Big Datainformation management and control system of claim 18 wherein said firstsensor network remote sensor station transmits periodic keep-alivesignals to said connected separate monitor unit to assure users that theremote sensor station is operating correctly when said broadcastedbackground audio signals are removed from or attenuated in signalstransmitted to a connected, separate monitor unit to minimize annoyingor unnecessary signals received and/or heard at said separate monitoringunit while still transmitting audio signals from said monitored objector person.
 20. The artificial intelligence IoT (Internet of Things) BigData information management and control system of claim 2 wherein saidsensor signals include a combination of at least one telecommunicationnetwork sensor input wherein said telecommunication network sensor inputmay originate from a signal quality sensor, modem sensor, transmissionlink sensor, router sensor, or switching system sensor.